Overview › 9Verification › c.Sizing verification

Sizing verification

Match verification effort to what it catches. 3 optimizations.

1. i.

   Don't prompt agents to write their own tests; suppress it for test-heavy models

   Agent-written tests are mostly print probes and do not change resolve rate.

   Lower costFaster

   Tier 3 · tuning · changes control flow · code generation agents · 6 sources, 2 measured

2. ii.

   Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays

   Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost).

   Better qualityHigher costSlower

   Tier 3 · tuning · changes control flow · any agent · 36 sources, 33 measured

3. iii.

   Audit a verifier-gated cascade against a gold sample the verifier never sees

   When a cheap model's outputs are accepted or escalated by a verifier, measure true error on a small gold-labeled sample outside the verifier: metrics measured through the verifier showed about 3% error while true error was 14-32%.

   Better qualityHigher cost

   Tier 4 · restructuring · changes control flow, model · any agent · 13 sources, 13 measured