7.c.iii Evaluator-optimizer loop

A separate evaluator (an LLM judge with written criteria, a different model, or an agent that tests the running app) scores the generator's output and returns specific findings; the generator revises until the output passes or a round cap is hit. It pays when the task is beyond what the generator does reliably in one pass and the evaluator is more reliable than the generator: a production diff judge vetoed about a quarter of coding-agent sessions, and half of those recovered. It fails when the judge is weak or shares the generator's blind spots: judge loops left table extraction worse than the first output and raised essay scores while human scores fell. Most of the gain comes in the first two to four rounds, at roughly twice the tokens of one call.

Tier 4 because it adds a generate-evaluate loop to the workflow.

When it applies

In the code: A generate, evaluate, revise loop: a separate evaluator call, model or tool-using agent scores the output against written criteria and returns feedback, with a pass threshold and a round cap. Flag an evaluator that shares the generator's dialogue history, one weaker than the generator, no cap, or a loop that returns the last version instead of the best.
In traces: Rounds per item; evaluator score per round next to an independent check; veto rate and recovery after a veto; items whose evaluator score rose while the independent check fell; revisions that rewrote parts that had passed.
In eval results: Quality per round on a grader the loop never sees; share of items that end worse than their first output; evaluator agreement with human or gold labels.
With a judge: The evaluator is a judge; a meta-judge checks its agreement with humans.
With probes: Items with a clear rubric and a known defect.

How

Start from the single pass, tuned with the same effort, and from best-of-N at equal tokens; add the loop only where outputs often fall short and no compiler, test or execution check can judge them (if one exists, feed its verdict back instead, 9.a.i).
Write the evaluator's criteria as a few gradable items, each with a hard pass threshold, and calibrate it with a few scored examples; where possible let it check by acting (drive the app with Playwright, compare the diff with the original request). The same criteria in the generator's prompt also steer the first draft.
Make the evaluator separate and skeptical: its own call and context, not the generator's dialogue history; a model at least as capable as the generator; tune its prompt by reading its logs against your own verdicts until it stops approving problems it found. Check it on labeled good and bad outputs and planted-defect probes before wiring it in. A judge from another model family does not remove favoritism (judges favor any model whose mistakes resemble theirs), and content-free answers such as ':' or 'Thought process:' can pass it, so include them among the probes.
Return findings the generator can act on (failing criterion or checklist item, where, expected against actual; told which checklist items failed, Gemini 2.5 Pro went from 29.5 to 94.7, against -2.5 to +2.6 with no feedback) and tell it to keep what already passed; unconstrained regeneration breaks parts that were right.
Stop on pass or after a low cap (most gains came in the first 2 to 4 rounds); raise the cap only if the per-round curve keeps rising. Return the best version rather than the last, and stop early when the score stalls or drops.
Measure against a check the loop never sees (a human-labeled sample or a deterministic metric), per round: quality, share of items that end worse than their first output, evaluator agreement with the audit, tokens and latency.

How to measure it

experiment (tier 4): paired eval of the loop against the single pass and against best-of-N at equal tokens, per model, graded by a check the loop never sees (human-labeled sample or deterministic metric): quality per round, share of items ending worse than their first output, tokens, latency
monitor: rounds per item, first-pass rate, veto rate and recovery rate after a veto, cap-hit rate; on a regularly audited sample, evaluator verdicts against human labels and drift between evaluator scores and the audit

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

Quality: mixed, medium: gains where the generator falls short and the critic is reliable: CodeContests 7.88 to 16.24 with a trained critic over five rounds, 12.48 with GPT-4o as critic (Xie, Feb 2025); Self-Refine +4.6 and +1.9 points over a tuned single call, -0.3 on chess (Leins, Aug 2026); losses where the judge is weak: table extraction 0.792 to 0.772 TEDS after 8 rounds (Kim, Jul 2026), essay judge scores rose while human scores fell (Pan, Jul 2024)
Cost: degrades, medium: an evaluator call plus a revision per round: Self-Refine took 1.82 to 2.13 times the tokens of a tuned single call (Leins, Aug 2026); in one app build the QA rounds cost $3.24, $3.09 and $4.06 of $124.70, and the rebuilds they triggered $36.89 and $5.88 (Rajasekaran, Mar 2026)
Performance: degrades, large: serial rounds; an evaluator that clicks through the running app took 6.8 to 9.6 minutes per round, and a 5 to 15 round design loop ran up to four hours (Rajasekaran, Mar 2026)
Security: none (unmeasured)
Governance: improves, small (unmeasured): Criteria are explicit.
Brand: improves, medium (unmeasured): Polished output.

Caveats

Most studies are small or single-run (one app per harness, 150 Bash questions, 23 essays), and the largest critic gain came from a critic trained for its generator, which is closer to 7.c.ii; prompted critics, GPT-4o included, helped less and over five rounds cut MBPP+ from 77.83 to 74.60. On outputs with no deterministic check the evaluator's own error decides the sign: judges tied, inflated scores and were gamed when they shared the generator's blind spots or context. Gains shrink as generators improve: Anthropic found the evaluator unnecessary overhead for tasks within the newer model's reach, and in one study the loop helped only the weakest of six generators. The 2 to 4 round finding comes from a setup where ground truth decided which answers got feedback, so it describes a best case.

Evidence

Evidence comes from: Code generation, Reasoning, Document extraction, Judging.

Lists this practice in its agent-optimization pattern library (Workflow Patterns group)Not measured

Names the practice as a known agent optimization; reports no measured effect for it.

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Table 6; pattern library file master-agent-optimization-cookbook.md
Names the evaluator-optimizer workflow: one call generates, another evaluates and gives feedback in a loopNot measured

"one LLM call generates a response while another provides evaluation and feedback in a loop"; "particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value."

Anthropic (Erik Schluntz, Barry Zhang), "Building effective agents" · Dec 2024Older evidence · Workflow: Evaluator-optimizer
A separate evaluator agent with written criteria, few-shot calibration and the Playwright MCP grades a generator's frontend or full-stack app and sends its critique back for the next round

"tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work". Out of the box the evaluator would "identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway"; it took several rounds of prompt tuning. Design loop: 5 to 15 iterations; scores "improved over iterations before plateauing", and the author "regularly saw cases where I preferred a middle iteration over the last one". App build on Opus 4.6: QA rounds cost $3.24, $3.09 and $4.06 of a $124.70 run and caught display-only and stubbed core features; the evaluator "is worth the cost when the task sits beyond what the current model does reliably solo."

Prithvi Rajasekaran (Anthropic), "Harness design for long-running application development" · Mar 2026 · Why naive implementations fall short; Frontend design: making subjective quality gradable; Running the harness; Removing the sprint construct; Results from the updated harness
After the build, format and test verifiers pass, an LLM judge reviews the diff against the original prompt; a veto sends the coding agent back to fix its change before a PR opens

"out of thousands of agent sessions, the judge vetoes about a quarter of them. When that happens, the agent is able to course correct half the time." The most common trigger is "the agent going outside the instructions outlined in the prompt". "We have yet to invest in evals for our judge."

Max Charas, Marc Bruggmann (Spotify), "Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)" · Dec 2025 · Using LLMs in the verification loops
A separate critic gives feedback and the generator revises, for one or five rounds; the generator's own critique, GPT-4o as critic, and a critic trained with RL (CTRL) comparedMixed

Qwen2.5-Coder generator, Pass@1 averaged over CodeContests, LiveCodeBench and MBPP+: zero-shot 38.75; after five critique rounds 38.30 with its own critique, 39.98 with GPT-4o, 42.77 with the trained critic. CodeContests 7.88 to 12.48 (GPT-4o critic, five rounds) and 16.24 (trained critic); trained critic after one, two and three rounds 11.76, 14.18, 15.15. MBPP+ fell from 77.83 to 74.60 after five GPT-4o critique rounds. GPT-4o generator: 43.53 zero-shot, 44.13 after five rounds of its own critique, 45.83 with the trained critic. In an inference-only size sweep, "critics less capable than their generators typically degrade performance".

Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, "Teaching Language Models to Critique via Reinforcement Learning" · Feb 2025Older evidence · Table 1; Table 3; Table 4 and "The Effect of Generator Ability" (§4.4)
An LLM judge scores essays against a four-item rubric and gives feedback, an LLM author revises, for five rounds; same model in both roles with separate prompts; expert humans score every version offlineAgainst

With GPT-3.5, "the Online LLM Judge scores the model-edited essays much higher compared to the ground-truth Human scores", and "the Human scores demonstrate a decrease in quality in the last iteration"; an offline judge without the loop history tracked the human scores. The divergence appeared when judge and author saw the same dialogue history; with different histories "the gap between the Online LLM Judge and Human scores becomes statistically insignificant". Only Conventions kept improving for humans; on the other items human-judged improvement "either plateaus after a single iteration or even degrades". GPT-4 showed less divergence.

Jane Pan, He He, Samuel R. Bowman, Shi Feng, "Spontaneous Reward Hacking in Iterative Self-Refinement" · Jul 2024Older evidence · §3.3-3.5; §4; Figures 3-5
A separate feedback model that sees the ground-truth solution critiques each wrong answer; the solver retries for up to 10 roundsMixed

"The performance typically improves rapidly through the first 2-4 iterations before significantly slowing down", and every model plateaus below the accuracy it would reach if it used all the feedback. Feedback resistance accounted for "62.8-100% of errors" left after 10 rounds. Richer feedback mattered: with strong-model reflective feedback Llama-4-Maverick reached 73.3% on AIME and 96.5% on GPQA, "improvements of +26.7% and +10.6% over binary feedback". Models "claim to understand feedback and express willingness to update their beliefs (>95%) yet fail to actually incorporate corrections".

Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, Daniel Khashabi, "Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback" · Jun 2025Older evidence · §2.1; §3.2; Figure 3; Figure 4; Table 1
A reference-free LLM judge scores each table extraction and feeds its error claims back for 8 rounds of regeneration; judge scores also used to pick the best candidate or to accept revisionsAgainst

FinTabNet TEDS fell from 0.792 to 0.772 after 8 rounds (p<10^-7), 44% of the drop at the first regeneration; 41.6% of tables worsened and 23.9% improved; OmniDocBench +0.002, not significant. The loop's judge tied on 49.1% and 37.5% of distinct candidate pairs; cross judges from other families (GPT-5.4, GPT-5.4-nano, Claude Opus 4.6) tied less but "still showed near-zero or negative TEDS rank correlations", and on FinTabNet "the trivial always-iter0 baseline beats every judge policy". A loop with the judge's feedback removed degraded by the same amount (99 paired tables). Across six generators the loop helped only Claude Haiku 4.5, whose first outputs were weakest (+0.0917 from 0.5675); Spearman 0.771 between headroom and net effect (n=6).

Donghwan Kim, "LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition" · Jul 2026 · Abstract; §4.1; §4.2; Appendix E Tables 19-20
Self-Refine with a separate critic call ("Act only as a critic of the proposed solution ... Do not rewrite the solution"), at most two feedback-refinement rounds, stopping when the critic says no refinement is needed, against a single chain-of-thought call; all prompts tuned with GEPA under the same budgetMixed

Accuracy, Self-Refine against optimized CoT: Codeforces 66.1% vs 61.5% (+4.6, 95% CI [2.1, 7.1]); AMC 89.8% vs 87.9% (+1.9, [0.5, 3.3]); Lichess 30.7% vs 31.0%. Weighted tokens 2.12, 2.13 and 1.82 times CoT. Benefits "did not increase systematically with difficulty", and "additional computation has little value when the backbone cannot generate a viable solution".

Nicolas Leins, Nico Pelleriti, Jana Gonnermann-Müller, Sebastian Pokutta, "When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty" · Aug 2026 · §3.3; §4.2; §4.5; §5; Appendix B.4
A dedicated evaluator scores generated Bash with two LLM-as-judge checks (bidirectional functionality matching, logic representation) and sends its judgment and feedback to the generator; against no refinement and against self-reflection

Execution accuracy on three suites, baseline / self-reflection / dedicated evaluator: GPT-4o 86/84/88%, 72/76/82%, 68/72/74%; Llama-3.3-70B 68/74/72%, 52/64/76%, 46/52/50%; Mistral-large 76/76/80%, 66/70/76%, 54/58/58%. The evaluator loop beat no refinement in all nine cells and self-reflection in six; it helps "but not always; it still depends on the specific model and dataset".

Ngoc Phuoc An Vo, Brent Paulovicks, Vadim Sheinin, "LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation" · Jun 2025Older evidence · §4.2; §5; Table 1
A Reviewer grades each game version from trajectories a coding-native Player collects, plus screenshots, against a game-specific rubric, keeps the better version and sends revision feedback to the Designer and Builder; three rounds

GameCraft-Bench overall: same-model baseline 71.26; rounds one, two and three 72.70, 75.35, 77.89. Action stayed below the baseline's 73.33 for two rounds (67.96, 70.33) before 77.62 in round three. GameASG-Bench task success 25/47 (53.2%) against 9/47 (19.1%) for the same model in Codex CLI, a baseline taken from another paper.

Jiajun Chen, Haoyu Wu, Mingda Jia, Xihui Liu, "Recursive Game Creator: An Agentic Product-Level Experience-Oriented Game Harness" · Oct 2026 · §3.3; §4.1.1; §4.1.2; Tables 1-2
Measure how a judge's scores depend on how similar the judged model's mistakes are to the judge's own (CAPA), controlling for accuracy, and whether frontier models' mistakes are convergingAgainst

"LLM-as-a-judge scores favor models similar to the judge, generalizing recent self-preference results"; "model favoritism extends beyond self- or family- preference to all models that are functionally similar". Example: Qwen2.5-72B-Instruct as judge scored Qwen2.5-7B-Instruct 71% correct and the more capable Llama-3.1-70B-Instruct (51% against 41% MCQ accuracy) 67%. In multiple regression both accuracy and similarity had a significant positive effect on the judge score; "model mistakes are becoming more similar with increasing capabilities".

Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, Jonas Geiping, "Great Models Think Alike and this Undermines AI Oversight" · Feb 2025Older evidence · §3.2 (Q1, Q2), Figure 3, Table 2; §5.2, Figure 6
A model answers, judges its own answers without a reference and is trained by DPO on the verdicts; a held-out exact-match or unit-test anchor the judge never sees audits the result; defenses compared (recompute prompt, larger and other-family judges, ensembles, committing an answer before judging, solving blind)Against

Full GSM8K test set, Qwen3-4B: "self-play drives the judge’s pass rate from 0.716 to 0.938±0.016 while anchor accuracy stays at 0.209→0.202±0.005"; "the judge reports 94% correct when 20% are". Under natural chain-of-thought (base accuracy 0.76-0.88) the gap fell to 0.086-0.125. Training against a different-family judge left a gap of 0.40±0.09; a three-family ensemble that accepts only when all agree "still accepts 55% of full-test wrong answers" on seed 0, 65% averaged over three seeds. Making the judge write its own answer first cut the false-positive rate on wrong answers "from the verify prompt’s 0.719 to 0.012"; used as the training reward, the blind-solve verdict accepted 0 of about 380 wrong answers. Best-of-16 selection by the self-judge on LiveCodeBench widened the gap from 0.20 to 0.588 while unit-test pass of the selected candidates went from 0.27 to 0.29.

Chenyu Zhou, "More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges" · Jul 2026 · Abstract; §5.1; §5.2; §5.3
Probe general-purpose and trained generative reward models with content-free 'master key' answers (punctuation, reasoning openers) in reference-based grading; retrain judges with 20k truncated outputs added as negativesAgainst

"such simple hacks lead to false positive rates (FPRs) as high as 80%"; "Thought process:" led to "FPRs as high as 60-90%" in LLaMA3-70B-Instruct and Qwen2.5-72B-Instruct, and ":" to "up to 35% FPRs" in GPT-4o; the trained General-Verifier "shows an alarming FPR of 66.8% on the MATH dataset using a naive single blank space". It surfaced when an RLVR policy trained against a Qwen2.5-72B-Instruct judge collapsed into short reasoning openers. Adding 20k first-sentence truncations labeled NO to 160k examples gave Master-RMs "near 0% FPR" on unseen master keys, with Cohen's kappa 0.91 against GPT-4o and 0.90 against humans for the 7B. The authors "confirm the ineffectiveness of common inference-time techniques such as chain-of-thought and majority voting".

Yulai Zhao, Haolin Liu, Dian Yu, Sunyuan Kung, Meijia Chen, Haitao Mi, Dong Yu, "One Token to Fool LLM-as-a-Judge" · Jul 2025Older evidence · §1; §3.2; §3.3.1 Table 1; §3.3.2 Table 2
Up to five turns of guided refinement (told which checklist items failed) against self-refinement (asked to improve its own answer with no feedback, deciding when to stop) and against self-refinement given the full checklist; each answer graded item by item by GPT-4.1

Pass (all checklist items met), turn 1 to turn 5, guided refinement: Gemini 2.5 Pro 29.5 to 94.7, Claude-Opus-4.1 18.7 to 98.4. Self-refinement: Gemini 2.5 Pro 29.5 to 31.3; GPT-5 27.5 to 29.1; o1 18.5 to 18.3; DeepSeek-R1 8.1 to 7.9; across all models in Table 3 the change ranged from -2.5 to +2.6. Given the full checklist but no feedback, "LLaMA-3.1-70B-Instruct improves by +43.6% and Gemini 2.5 Pro by +44.5%" at turn 5 over plain self-refinement.

Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi, "RefineBench: Evaluating Refinement Capability of Language Models via Checklists" · Nov 2025 · §4.2 Table 3; §5.1 Table 4; §5.3 Figure 9

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:evaluator-optimizer-loop. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.

Evaluator-optimizer loop