Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays

Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost). Before adding a generative verifier on top, compare it with the vote at equal compute: it needs 4-64x the compute to match, and pays only on hard, medium-difficulty problems from weak-to-medium generators. If you do verify, an ensemble of diverse weak verifiers weighted by estimated accuracy beat one strong verifier (87.7% against 72.2%).

Tier 3 because extra compute within the existing design.

When it applies

• In the code: a single attempt per question; or a generative verifier or best-of-N step with no comparison against a plain vote
• In traces: answer disagreement across repeated runs of the same question
• In eval results: accuracy of vote against verifier at equal compute, split by difficulty

How

1. Measure variance across repeated runs on the same questions; if low, stop here.
2. Add N parallel attempts with a vote or aggregator; paired eval at matched compute. At a fixed thinking budget, parallel traces with a majority vote beat forcing one trace to think longer.
3. Only then try a verifier, at the same compute, on the hard slice.

How to measure it

• experiment (tier 3): paired eval, N attempts against one, and vote against verifier, at matched compute

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: improves, medium: Genie +7.6 points with 8 parallel runs (May 2026); verifier over vote: gains only on hard problems (2025)
• Cost: degrades, large: 0.23x to 2.13x cost; verifier ensembles far more
• Performance: degrades, small

Evidence

Evidence comes from: Reasoning, Data analysis, Planning.

1. Prefer self-consistency over generative verification at low budgets

   Llama-3.1-8B GenRM-FT on MATH: GenRM needs 8x the compute of SC to match it, and 128x to beat SC's best by 3.8%; SC plateaus around 128 solutions. Qwen-2.5-7B GenRM-FT: SC peaks with 64x less compute; GenRM +5.4% at 512x. Llama-3.3-70B GenRM-Base: SC peak at 4x less; GenRM +1.7% at 64x. QwQ-32B on AIME25: GenRM needs 4x to match, +2.5% at 16x. GPQA-Diamond (70B): +2.5% at 256x. LiveCodeBench (8B): 16x to match, +3.5% at 64x. Comparing at fixed solution count instead (GenRM matches SC with 4x fewer solutions) is called misleading.

   Singhi et al., "When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning" · Apr 2025Older evidence · §5.1, Figures 1, 4, 5, 8, 9; Appendix G, H

2. Reserve generative verification for hard problems

   Llama-3.1-8B GenRM-FT (32 verifications) vs SC: up to 30% relative improvement on MATH level 5 vs much less on level 1. AIME24 (trained on MATH): GenRM almost doubles SC's performance but needs 256x more compute; SC reaches its peak with 16x less compute than GenRM needs for the same level.

   Singhi et al., "When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning" · Apr 2025Older evidence · §5.1 Impact of Problem Difficulty, Easy-To-Hard Generalization; Figures 3a, 4a

3. Scale solutions faster than verifications per solution

   Fitted power laws on MATH: Llama-3.1-8B GenRM-FT S_opt ~ C^0.57, V_opt ~ C^0.39; Qwen-2.5-7B GenRM-FT S_opt ~ C^0.75, V_opt ~ C^0.32; Llama-3.3-70B GenRM-Base S_opt ~ C^0.69, V_opt ~ C^0.43. Up to 128 solutions x 128 verifications sampled.

   Singhi et al., "When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning" · Apr 2025Older evidence · §3.2, §5.2 Figure 6; Appendix J Figures 11-13

4. Shorten verification outputs to cut verifier cost

   Analytical upper bound assuming constant verifier accuracy (Llama-3-8B GenRM-FT, MATH): tokens per verification 2048 -> compute to match SC 8x, to beat 128x; 1024 -> 4x/64x; 512 -> 2x/32x; 256 -> 1x/16x. The authors note keeping accuracy while shortening is non-trivial.

   Singhi et al., "When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning" · Apr 2025Older evidence · Appendix F, Table 1

5. Expect single generative verifications to be noisy

   "In most cases, at least 32 verifications are needed to achieve good results" (Llama-3.1-8B, MATH).

   Singhi et al., "When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning" · Apr 2025Older evidence · Appendix E

6. Skip strong verifiers on strong generators

   Math: Qwen2.5-72B verifier TNR 0.68 on Llama-3.1-8B solutions vs 0.17 on Qwen3-32B solutions. With strong generators, verification gain peaks at only about 0.1 for both GPT-4o and Qwen2.5-7B verifiers and the gap between them narrows. Holds for thinking verifiers (Qwen3-8B/32B-Thinking), Qwen3-235B and three verification prompts.

   Zhou et al., "Variation in Verification: Understanding Verification Dynamics in Large Language Models" · Sep 2025Older evidence · §4.2 Figure 4; §5.2 Figure 8; Appendix C.2, C.6 Figure 21, D.3, D.4

7. Skip verification on the easiest and hardest problems

   Math: verification gains drop to 0.1 or below for both GPT-4o and Qwen2.5-7B on easy and hard problems. On hard problems (difficulty [0.1,0.3)) verification accuracy plateaus around 0.65 in math regardless of verifier capability and is flat elsewhere; on hard 3-way NL reasoning verifiers score below random because they solve-and-match with a wrong answer. Medium problems ([0.4,0.5)) show linear capability-accuracy correlation (r>0.9).

   Zhou et al., "Variation in Verification: Understanding Verification Dynamics in Large Language Models" · Sep 2025Older evidence · §4.3 Figure 6; §5.2 Figure 8; Appendix C.3, C.4, C.6 Figure 21

8. Estimate difficulty from answer entropy of a small model

   8 samples from Qwen2.5-3B per question, difficulty = H/log K. Reproduces the oracle-difficulty findings for TPR vs difficulty and capability-accuracy regimes (math). No downstream accuracy or cost number reported.

   Zhou et al., "Variation in Verification: Understanding Verification Dynamics in Large Language Models" · Sep 2025Older evidence · Appendix D.1, Figures 22-23

9. Do not scale verifier size to fix hard-problem verification

   39.1% of verification false negatives on hard problems contain a mistake in the verifier's own reference answer (LLM-judge analysis). Hard [0.2,0.3) math: Qwen3-32B (capability ~0.6) and Qwen2.5-72B (~0.4) both plateau near 0.7 verification score. "Scaling verifiers from 7B models to GPT-4o fails to overcome fundamental verification challenges" in these regimes.

   Zhou et al., "Variation in Verification: Understanding Verification Dynamics in Large Language Models" · Sep 2025Older evidence · §4.1 Figure 3; §5.2; Appendix C.4

10. Prefer self-consistency voting over multi-agent debate

    Table 7, GSM8K full test set, gpt-3.5-turbo-0301, 3 agents, Du et al. prompts: standard 76.7 (1 response); self-consistency 82.5 (3); debate round 1 83.2 (6) vs self-consistency 85.3 (6); debate round 2 83.0 (9) vs self-consistency 88.2 (9).

    Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" · Oct 2023Older evidence · Sec. 4; Table 7

11. Compare self-correction with self-consistency at equal computeNot measured

    Survey RQ3: self-correction is often not compared to strong baselines, so it is still unclear whether it beats them; cites self-consistency and pass@k comparisons.

    Kamoi et al., "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs" · Jun 2024Older evidence · Sec. 6; Table 7 checklist

12. Select among many samples by self-verification, not votingMixed

    Gemini 1.5 Pro, k_inf=200, k_verif=50, with tie-break: AIME 8/15 vs Consistency@200 4/15 and Pass@1 1/15 (o1-Preview reported 7/15); MATH 467/500 vs 460; LiveBench Math 135/200 vs 118; LiveBench Reasoning 97/140 vs 75. Consistency@1,000 no better than @200 (e.g. AIME 3/15). Single runs.

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §1 Table 1; Algorithm 1; §3.1

13. Keep sampling past self-consistency saturation when a verifier existsMixed

    AIME: Consistency@50 equals Consistency@10,000 while Verification@k (k_verif=50) shows power-law scaling in k. Example AIME 2024 II P11: 1 of 200 samples correct (answer 601, verification score 0.98) vs majority answer "1" from 124 samples (score <=0.36). Accuracy on "ambiguous" questions (at least one correct candidate) rises with k at fixed k_verif (implicit scaling). Largest gains when both k_inf and k_verif grow; strongest on AIME where correct answers are rare.

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §2.1 Figure 2.1; §2.2 Figures 2.2-2.3; §2.3 Figure 2.4, Table 2

14. Break verifier ties by pairwise comparison of candidates

    Gemini 1.5 Pro, k_tie=100 per matchup, Verification@200 without vs with tie-break: MATH 457 -> 467/500 (Cons@200 460); LiveBench Math 125 -> 135/200 (Cons 118); LiveBench Reasoning 94 -> 97/140 (Cons 75); AIME 7 -> 8/14 (Cons 4). Helps most when the verifier over-assigns high scores (MATH top-3 average score near 90%).

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §3 principle 1; §3.1 Tie-Break; §3.2 Table 3

15. Have the verifier rewrite candidates as theorem-lemma-proof first

    Verification prompt ablation, Gemini 1.5 Pro, 20 verification attempts per response, 1,080 responses to 54 MATH level-5 train questions and 120 responses to 6 AIME 2023 questions: Main (with rewrite) FPR/FNR MATH 14%/17%, AIME 7%/7%; Without Rewrite MATH 16%/18%, AIME 11%/12%. Fails where the answer format does not fit a theorem statement (LiveBench Olympiad: Verif@200 22.7% vs Cons@200 25.0%).

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §3 principle 2; §3.2 Table 4; §4.2 Table 6; Appendix B.2

16. Do not split verification into separate per-piece threads

    Same ablation as the rewrite row: Split-Context FPR/FNR MATH 19%/23%, AIME 11%/14% vs Main 14%/17% and 7%/7%; attributed to miscalibration.

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §3.2 Table 4

17. Prune candidates with cheap preliminary verification scores

    10 preliminary scores, discard average <0.2, keep top 15 per final answer, then 50 scores on the rest: cuts output tokens about 70%, per-question cost about $650 -> $200 (Pro); Flash as verifier brings it to about $12. Accuracy impact of pruning not separately reported. Used on all datasets except AIME.

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §5 Preliminary scoring, Compute

18. Do not trust out-of-box frontier-model verification

    Curated verification benchmark (random-guess baseline: 80% correct, 20% wrong, 20% flawed, 50% comparison). GPT-4o Pass@1: 76.5% / 31.0% / 22.2% / 43.2%; Claude 3.5 Sonnet 89.6% / 22.5% / 33.3% / 56.1%; o1-preview 100% / 68.8% / 80.0% / 84.5%. Consistency@5 lowered GPT-4o on flawed (22.2 -> 11.1) and comparison (43.2 -> 35.4).

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · §6 Table 7

19. Sampling temperature barely matters for verified search

    MATH train split, 100 questions, 20 samples, Verification@20 without tie-break: inference temp 0.2/1.0/1.5/2.0 -> 82/80/79/82; verification temp 0.2/1.0/2.0 -> 76/79/79.

    Zhao et al., "Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification" · Feb 2025Older evidence · Appendix B.1 Tables 8-9

20. Expect verifier ensembles to cost far more than voting

    Majority voting plateaus around 2^2-2^3 ExaFLOPs per query while Weaver keeps improving up to 512 ExaFLOPs (K from 1 to 100); the authors call the extra compute "prohibitive". Standard baselines (majority vote, single RM, self-verification, multi-agent verification) plateau after a few generations; self-verification averaged 66.9% and multi-agent verification 71.6% at K=100, below majority vote (72.2%).

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · §5.2 (1) Figure 3, (4) Figure 5; Table 1

21. Weight verifiers by estimated accuracy, not naive averaging

    Llama 3.3 70B, K=100, average of 4 benchmarks: Weaver 87.7% vs majority vote 72.2%, best single RewardBench RM 72.7%, naive top-10 RM ensemble 69.1%, first sample 68.4%, Pass@100 oracle 91.9%; o3-mini first sample 86.7%. Oracle-labeled weighted ensembles beat naive averaging by up to 11.2 points (3.6 for top-K selection, 7.8 for supervised weights on average). Individual verifier success rates span up to 37.5 points.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · §4.1 Figure 2; §5.1 Table 1; Appendix Table 16

22. Estimate verifier accuracies from agreement with weak supervision

    Supervised logistic regression / naive Bayes weights fit on 0.01n labeled samples lose 20.1% on average vs their oracle versions, which motivates the label-free estimate. Weaver beats naive averaging by +2.4 to +10.1 points across datasets (GPQA +10.1, MMLU Pro +5.1) with 1-15 verifiers.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · §4.2 eqs 1-5; §5.2 Figure 4; Appendix B.1, Table 17

23. Drop low-signal verifiers before ensembling

    Verifier pruning gives a 12.5% performance improvement in the 70B setting; also decorrelates the verifier set (precision-matrix off-diagonals attenuate).

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · Appendix B.3, Table 5, Figure 14

24. Binarize verifier scores with dev-set thresholds

    Dev-set thresholds from 5-10 labeled queries (<=1% of eval) improve Weaver by 8.4% on average for 70B vs median-split binarization. Binarization strategies differ only modestly in selection accuracy; quantile (top 15%) does best on imbalanced GPQA. With ample labels, continuous-score logistic regression beats discrete.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · Appendix B.2-B.3, Table 4, Table 7, Figures 8-9

25. Add more verifier models rather than resampling one

    Llama 3.3 70B generations: adding more models gives average gains of 5.4% (RMs) and 6.1% (LM judges) vs 0.8% and 1.1% from extra scores from a single RM or judge. Aggregating up to 15 verifiers improves up to 8.5% over the top-1 verifier, with diminishing returns from correlated biases on hard examples.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · §5.2 (3); Appendix C.5 Table 21

26. Distill the verifier ensemble into a small cross-encoder

    GPQA Diamond, Llama 70B generations, 80:20 split: retains 98.2% of Weaver's accuracy (98.7% stated elsewhere); 1.01 vs 35.35 exaFLOPs per 100 samples (99.97% verification FLOPs saved); beats majority vote by 23.2% at 0.57% more inference cost than generation alone. Runs on one A100 instead of an 8-GPU node per 70B verifier.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · §6 Figure 6; Appendix C.6 Figure 24, Table 22

27. Cluster queries by difficulty for per-cluster verifier weights

    Oracle difficulty clusters (1-5) with threshold grid 0.05-0.95: 8B setting +4.8%; 70B about +1% and more clusters often degrade it. Per-model thresholds add +0.5 to +0.8 (70B) and +1.6 to +2.8 (8B).

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · Appendix B.4, Tables 9-11

28. Optimize LM-judge prompts for precision

    250 labeled examples, DSPy 0/3/5-shot: average +3.8% precision of best judge over CoT baseline, MATH500 +9%; scaling training data 1% -> 16% gives about +3.2% precision on most datasets (MMLU-College minimal). Gains come mainly from lower false-positive rates.

    Saad-Falcon et al., "Shrinking the Generation-Verification Gap with Weak Verifiers" · Jun 2025Older evidence · Appendix C.7 Figures 25-27

29. Decorrelate the verifier from the student modelNot measured

    Stated as a design rule from Corollary 1 (floor increasing in beta0; same-family verifiers should give higher floors). Not directly measured on real models; the synthetic study measures floor vs beta0 (0.09 to 0.36 as beta0 sweeps 0.19 to 0.96), not family correlation.

    Rajput et al., "Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades" · Sep 2026 · §6 Corollary 1; §7 H2 Figure 7b

30. Reject on any dissent across a decorrelated verifier ensemble

    Synthetic model only (rho=0.7): measured beta0 0.76 -> 0.40 at m=4 (above the rho^4~0.24 ideal); floor 0.25 -> 0.14; cost m verifier calls per query. Exchange rates are "properties of the synthetic geometry, not portable constants".

    Rajput et al., "Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades" · Sep 2026 · §7 H5, Figure 8; §9 limitation 5

31. Verify by consensus of partial criticsNot measured

    Proposal in the conclusion; not evaluated here.

    Stechly et al., "On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks" · Feb 2024Older evidence · Sec. 6

32. Sample parallel trajectories and aggregate with a judge

    Internal benchmark, Figure 5: GPT-5.4 65.5% (3.8 min avg runtime) -> 75.7% (8.2 min) with parallel thinking (+10.2pp); all Opus 4.6 73.8% (5.5 min) -> 81.4% (9.5 min) (+7.6pp). Figure 1: N=8 with Opus 4.6 raises relative cost 0.23x -> 2.13x. Error bars shown but n not stated.

    The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Parallel Thinking section; Figures 1 and 5

33. Score partial plans by progress, bottleneck and potential

    Table 2, RealEHR, GPT-5.4-mini: removing plan evaluation 55.2 -> 44.7 (-10.5pp), the largest ablation.

    Youngwon Lee, Jaejin Kim, Seung-won Hwang (Seoul National University), "Bootstrapping Semantic Layer from Execution for Text-to-SQL (GATE: Grounding After Test from Execution)" · Jun 2026 · §3.3 Plan evaluation and selection; §4.3; Table 2

34. Pick the next bottleneck that is testable and informative

    Table 2, RealEHR, GPT-5.4-mini: removing plan selection 55.2 -> 53.2 (-2.0pp, about one question of 47).

    Youngwon Lee, Jaejin Kim, Seung-won Hwang (Seoul National University), "Bootstrapping Semantic Layer from Execution for Text-to-SQL (GATE: Grounding After Test from Execution)" · Jun 2026 · §3.3; §4.3; Table 2

35. Search structured plans instead of repeating independent runs

    Table 3, RealEHR, GPT-5.4-mini, avg LLM calls per query: ReAct 42.9% at 6.3 calls; ReAct x6 with the same selector 51.1% at 39.1 calls; GATE 55.2% at 36.0 calls (+4.1pp, fewer calls).

    Youngwon Lee, Jaejin Kim, Seung-won Hwang (Seoul National University), "Bootstrapping Semantic Layer from Execution for Text-to-SQL (GATE: Grounding After Test from Execution)" · Jun 2026 · §4.4; Table 3; Appendix B.6

36. Spread a fixed thinking budget over parallel traces with a majority vote, not one longer trace

    DeepSeek-R1-Distill-Qwen-1.5B on GSM8K: "pushing the average thinking token count from 1100 to 15980 reduces accuracy from 87.3% to 70.3%" with forced "Wait" extensions. At a 16,000-token budget, "Wait & Think More" gave "a 11.8% performance drop" against standard thinking, while parallel thinking with the same budget "boosts accuracy by 10.1%".

    Ghosal et al., "Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models" · Jun 2025Older evidence · §2.3 Figure 2; §4 Figures 6-7

Also known as

• Self-consistency voting
• Inference-time scaling
• LLM-as-judge

Related

• 9.c.iiiAudit a verifier-gated cascade against a gold sample the verifier never sees
• 7.c.iiiEvaluator-optimizer loop
• 9.b.iiTriage false success with a cheap trained detector, not an LLM judge
• 7.c.ivGraph-of-Thought
• 7.c.vLanguage Agent Tree Search (LATS)
• 1.b.iMatch the model and its effort to each call site and query class
• 7.c.viMulti-agent debate
• 6.a.iOracle-worker multi-model
• 7.c.iReflection loop
• 7.c.iiSelf-critique evaluator loop
• 3.c.iSelf-Discover reasoning structures
• 7.c.viiTree-of-Thought

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:samples-before-verification. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.