Overview › 9Verification

Verification

8 optimizations in 3 sub-categories.

a.Checking externally

Check work against sound external signals

i.

Feed back a sound external verifier's verdict; never the model's own critique

Run a check the model cannot fool on its output (compiler, tests, schema validation, execution, EXPLAIN on generated SQL, a solver) and feed the verdict back; a pass/fail verdict is enough, and resampling until the check accepts works about as well as detailed feedback.

Better qualityHigher costSlower

Tier 2 · reliability · changes control flow · any agent · 31 sources, 17 measured
ii.

Verify citations claim by claim for factual support

Extract claim-citation pairs deterministically, then check each: the link resolves, the source is relevant, and the source actually supports the claim.

Better qualityLower cost

Tier 3 · tuning · changes control flow, knowledge access · web research, retrieval agents · 8 sources, 5 measured

b.Confirming completion

Confirm the task is actually done before stopping

i.

Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit

State the required answer format in the prompt, and make the harness extract it robustly: strip markers, normalize numbers, detect refusals, and force a best-effort answer when the loop ends on a budget rather than returning nothing.

Better quality

Tier 2 · reliability · changes control flow, prompt · structured output agents · 4 sources
ii.

Triage false success with a cheap trained detector, not an LLM judge

Train a lightweight text classifier (TF-IDF with logistic regression) on 50-100 labeled runs to flag likely false successes for review.

Better qualityLower costFaster

Tier 2 · reliability · changes control flow · judging agents · 6 sources, 6 measured
iii.

Verify completion against the environment before the agent stops

Never take the agent's own report that it finished.

Better qualityHigher costSlower

Tier 2 · reliability · changes control flow · any agent · 22 sources, 9 measured

c.Sizing verification

Match verification effort to what it catches

i.

Don't prompt agents to write their own tests; suppress it for test-heavy models

Agent-written tests are mostly print probes and do not change resolve rate.

Lower costFaster

Tier 3 · tuning · changes control flow · code generation agents · 6 sources, 2 measured
ii.

Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays

Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost).

Better qualityHigher costSlower

Tier 3 · tuning · changes control flow · any agent · 36 sources, 33 measured
iii.

Audit a verifier-gated cascade against a gold sample the verifier never sees

When a cheap model's outputs are accepted or escalated by a verifier, measure true error on a small gold-labeled sample outside the verifier: metrics measured through the verifier showed about 3% error while true error was 14-32%.

Better qualityHigher cost

Tier 4 · restructuring · changes control flow, model · any agent · 13 sources, 13 measured

Try it for yourself

Paste this into Claude Code, or any coding agent, in your agent's repo to check it for these optimizations.

Read these agent optimizations (Verification), each with the signs that it applies:
- 9.a.i Feed back a sound external verifier's verdict; never the model's own critique: https://mahalsystems.ai/aot/optimizations/external-verifier-feedback.md
- 9.a.ii Verify citations claim by claim for factual support: https://mahalsystems.ai/aot/optimizations/verify-citations.md
- 9.b.i Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit: https://mahalsystems.ai/aot/optimizations/extractable-final-answer.md
- 9.b.ii Triage false success with a cheap trained detector, not an LLM judge: https://mahalsystems.ai/aot/optimizations/false-success-detector.md
- 9.b.iii Verify completion against the environment before the agent stops: https://mahalsystems.ai/aot/optimizations/verify-completion.md
- 9.c.i Don't prompt agents to write their own tests; suppress it for test-heavy models: https://mahalsystems.ai/aot/optimizations/no-self-written-tests.md
- 9.c.ii Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays: https://mahalsystems.ai/aot/optimizations/samples-before-verification.md
- 9.c.iii Audit a verifier-gated cascade against a gold sample the verifier never sees: https://mahalsystems.ai/aot/optimizations/audit-verifier-cascade.md
Then check whether any of them applies to the agent in this repository. Look for each one's signs under "When it applies" in its code, prompts and tools, and in any traces or eval results you can find. Tell me which apply, strongest first, with what you found for each. Don't change anything yet.

Verification

a.Checking externally

Feed back a sound external verifier's verdict; never the model's own critique

Verify citations claim by claim for factual support

b.Confirming completion

Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit

Triage false success with a cheap trained detector, not an LLM judge

Verify completion against the environment before the agent stops

c.Sizing verification

Don't prompt agents to write their own tests; suppress it for test-heavy models

Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays

Audit a verifier-gated cascade against a gold sample the verifier never sees

See also

Triage false success with a cheap trained detector, not an LLM judge

Build better agents, weekly

You're on the list.