9.c.iv Write a fail-to-pass test before the fix, and harden it against wrong fixes

Make a coding agent's test a real check: write it before any source edit, admit it only if it fails on the unpatched code, then harden it against plausible wrong fixes or have a separate strong model write it and hold it fixed while the agent repairs. A test written together with the fix shares the fix's mistakes and adds little (58.9% to 61.4% resolved for GPT-5-mini); written first and hardened, it reached 69.4%, at $0.84 against $0.39 per issue. Frozen tests from GPT-5.6-sol raised it from 81.5% to 85.7% on SWE-bench Verified but only 0.7 points on SWE-bench Pro, and tests from a weak test writer lowered a repair agent from 61.2% to 57.3%.

Tier 3 because it changes what the agent does and what it spends, so quality and cost must be checked.

When it applies

In the code: a test step that runs after or alongside the fix, with no check that the test fails on the unpatched code
In traces: the agent's own test passing on a fix that the reference tests reject; tests that also pass on the unpatched code; runs that submit right after their own new test passes
In eval results: resolve rate with and without the test step at matched spend; for a separate test writer, the share of past issues where its test fails on the unpatched code and passes on the known fix

How

Start from the current agent as the baseline: resolve rate, cost and steps per issue with no test, and, for each check that passed after the agent's first edit, whether it also passes when replayed on the unpatched code (46.0% did for GPT-5.6-sol in one study).
Have the agent write the test before any source edit, run it on the unpatched code and reject it unless it fails there. Asserting the exact expected behavior matters more than printing values. In one study with GPT-5-mini, a test written together with the fix gave 61.4% resolved against 58.9% without, at $0.56 against $0.39 per issue; joint test and fix failures ran at 1.87 times the rate expected if they were independent, and 48.2% of unresolved runs had their own test passing on a wrong fix. Dropping the test-first phase cost 7.0 points.
Harden the test against plausible wrong fixes: run it against a few mutated versions of the fix (symptom suppressed, incomplete fix, input-specific shortcut) and strengthen it until they fail. Injected reproduction tests that also passed wrong fixes added nothing to the fix rate (+0.0 points) while strict ones added 8.5; the full loop took GPT-5-mini from 58.9% to 69.4%, at $0.84 against $0.39 per issue.
Alternatively, separate and freeze: a separate run writes the test from the issue, the harness admits it only if it fails on the unpatched code, and the repair agent then edits source only, with the test held fixed. First check the test writer on past issues: its tests should fail on the unpatched code and pass on the known fix (87.8% of SWE-bench Verified issues for GPT-5.6-sol, 61.0% for Codex-5.3, 22.2% for an untrained Qwen3.5-35B-A3B). Frozen tests from GPT-5.6-sol raised a Qwen repair agent from 61.2% to 65.3%, while tests from the untrained Qwen model lowered it to 57.3%.
A cheaper step on its own: have the harness tell the agent when a passing check also passes on the unpatched code (a median 11.0 s per run). It cut runs closing on such checks from 23.8% to 13.6% but did not change the resolve rate.
Measure with paired runs against the no-test baseline at matched spend where you can, several runs per arm: resolve rate, cost and steps per issue, and the share of kept tests that fail on the unpatched code and reject mutated fixes.

How to measure it

experiment (tier 3): paired runs against the no-test baseline, at matched spend where possible, several runs per arm: resolve rate, cost and steps per issue, and the share of kept tests that fail on the unpatched code and reject mutated fixes
monitor: in production traces: the share of the agent's own passing tests that also pass on the unpatched code, and fixes whose own test passes but whose review or reference tests fail

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use AOT Evals.

Expected impact

Quality: improves, medium: written first and hardened, GPT-5-mini 58.9% to 69.4% resolved (Tan, Jul 2026); frozen tests from GPT-5.6-sol 81.5% to 85.7% on SWE-bench Verified but +0.7 on SWE-bench Pro, and tests from a weak writer cut a repair agent from 61.2% to 57.3% (Tao, Sep 2026)
Cost: degrades, medium: $0.39 per issue with no test, $0.56 with a test written alongside the fix, $0.84 written first and hardened (GPT-5-mini; Tan, Jul 2026); the frozen-test gains are not compute-matched (Tao, Sep 2026)
Performance: degrades, small: the agent took more steps when told to write the test first (Cheng, Jan 2026); replaying passing checks on the unpatched code added a median 11.0 s per run (Xu, Jul 2026)
Governance: improves, small: code owners reviewing Google's AI-generated fixes strongly preferred fixes with a fail-to-pass reproduction test (Cheng, Jan 2026)

Caveats

Each measured gain comes from its authors' own method: CoHarden on 433 Python issues with one run per issue and no significance tests (with GPT-5.4 and Claude Opus 4.5 it was compared only with other test-writing methods), and ExecCritic, whose gain is not compute-matched and nearly vanished on SWE-bench Pro. A weak test writer makes results worse, so the writer must be checked first, and the quality gain costs about twice as much per issue. Apart from 120 internal bugs at Google, all results come from Python issue-resolution benchmarks.

Evidence

Evidence comes from: Code generation.

Have the same agent write a reproduction test with its fix (cogeneration) against fix only, then write the test before any source edit and harden test and fix over rounds against LLM-generated mutants of the fix

Table 2 (OpenHands with GPT-5-mini, 433 issues): fix only 58.9% resolved at $0.39 per issue; fix plus test in one run 61.4% at $0.56, its test failing on the buggy code and passing on the reference fix in 72.7%; CoHarden 69.4% at $0.84 (78.9%). Table 5: without the test-first phase 62.4%; without hardening 61.0% (82.6% fail-to-pass). §2.3: in 80 of 166 unresolved cogeneration runs (48.2%) the run's own test passed on its wrong fix; joint test and fix failure 19.9%, 1.87 times the 10.6% expected if independent. Table 1, separately generated tests injected into the fix agent: strict tests 67.3% -> 75.8% (+8.5, 552 pairs), lax tests that also pass wrong fixes 69.5% -> 69.5% (+0.0, 552), tests that fail the reference fix 44.5% -> 40.9% (-3.6, 474). Table 3: with GPT-5.4 and Claude Opus 4.5, CoHarden 76.2% and 78.7% against 70.0% and 70.8% for fix plus test in one run, at $6.69 against $4.26 and $18.07 against $13.99. A budget-matched 5-round self-refinement loop reached 63.8%.

Yuhao Tan, Zhibang Yang, Fangkai Yang, et al. (Nanjing University, Peking University, Microsoft), "Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes" · Jul 2026 · Abstract; §2.2 Table 1; §2.3 Figure 2; §3; §4.1 Table 2; §4.2 Tables 3-4; §4.3 Table 5; App. A.3; App. B.2
Have a separate test agent write a repository-native test from the issue, admit it only if it fails on the unpatched code, freeze it, and let the repair agent revise source from its results; vary who writes the test while holding the repair agent fixedMixed

§5.2.1: with the untrained Qwen3.5-35B-A3B repair agent fixed, no test 61.2% resolved; tests from the same untrained model 57.3% (-3.9); tests from GPT-5.6-sol 65.3% (+4.1); oracle fail-to-pass tests 69.4%. GPT-5.6-sol with generated tests: SWE-bench Verified 81.5% -> 85.7% (+4.2), oracle 89.7%; SWE-bench Pro 61.6% -> 62.3% (+0.7), oracle 73.4%. Table 1, tests that fail on the unpatched code and pass on the reference fix (one generation per issue): Qwen3.5-35B-A3B 22.2%, after SFT 39.6%, after RL 62.2%; Codex-5.3 61.0%; DeepSeek-V4-Flash 73.4%; GPT-5.6-sol 87.8%. Table 2: trained test and repair agents together 72.6% against 68.3% for the trained repair agent alone. Authors: "when the same trajectory writes both the patch and the test, their errors can agree and create false confidence"; the composed gain "reflects the full system, including additional test-generation and revision computation, rather than a compute-matched improvement."

Leitian Tao, Baolin Peng, Haorui Wang, et al. (University of Wisconsin-Madison, Microsoft Research), "ExecCritic: Learn to Test, Test to Improve for Coding Agents" · Sep 2026 · Abstract; §1; §2; §3.1-3.2; §5.1; §5.2.1 Figure 5; §5.2.2 Table 1; §5.2.3 Table 2; App. A.3
Instruct Google's production repair agent to return a reproduction test with its fix (in any order, test first, or test last) against fix only or test only, on internal bugsMixed

§5.1: all three cogeneration prompts "produce at least one plausible fix for a similar number of bugs compared to Fix-only" (pass@k plotted in Figure 2, not tabulated); the agent took more steps when told to prioritize the test. Table 2: told to write the test first, the agent edited a test first in 78.66% of runs (test-last 4.16%, freeform 24.54%, fix only 2.27%). Table 3: probability of a plausible fix given a test that fails on the bug and passes on the agent's fix, 0.53 test-first against 0.45 freeform and test-last. Table 4 (freeform): plausible fix in 0.14 of patches with no test, 0.24 with a test, 0.44 when the test fails on the bug and passes on the fix. §2.2: code owners reviewing the agent's fixes "strongly prefer fix patches to also have an accompanying fail-to-pass BRT"; 39% of human-written null-pointer fixes in 2024 changed a test.

Runxiang Cheng, Michele Tufano, José Cambronero, et al. (Google), "Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair" · Jan 2026 · §2.2.1-2.2.2; §3.1 Listings 1-3; §4.1-4.2; §5.1; §5.2 Tables 2-3; §5.3.1 Table 4
Replay every check a repair agent ran after its first edit (repository tests it chose, its own scripted oracles and print probes) on the unpatched code, its candidate and the developer fix, and test whether telling the agent when a passing check also passes on the unpatched code changes how it closes

Table 3: of 2,548 passing checks with evaluable replays (3,730 events in 643 runs), 1,141 were regression-only and 30 misleading, so 46.0% also passed on the unpatched code; 26.9% of the checks that failed on the unpatched code and passed on the candidate also failed on the developer fix. 23.8% of baseline runs submitted with only such non-discriminating passing evidence. Table 4 (preregistered, 110 tasks, 2 runs per arm): evidence-inadequate closure 23.8% baseline, 21.4% with a matched generic reminder, 13.6% with bug-contrast feedback (feedback minus reminder -7.8 points, 95% CI -12.9 to -2.7, p=0.0029); official resolution 65.9%, 66.0% and 66.8% (feedback minus reminder +0.48 points, CI -3.73 to 4.69). Table 5: the replay added a median 11.0 s per run; mean wall time 390, 381 and 440 s. Exploratory 20-task replications: baseline evidence-inadequate closure 40.5% for gpt-5.6-terra against 18.9% for gpt-5.6-sol, and 8.3% under a plan-execute-verify scaffold; on terra the reminder alone matched the full effect.

Xiaonan Xu, Wenjing Wu (Georgia Tech, University of Colorado Boulder), "Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?" · Jul 2026 · Abstract; §3.1-3.4; §4.2-4.3 Tables 1-2; §5.1 Table 3; §5.3-5.5 Tables 4-5; §5.6 Table 6; §6.3
Reserve test writing for bugs with explicit repro conditionsNot measured

Exploratory only: 8 SWE-bench Verified issues appear both in encourage-prompt Fail->Success-with-new-tests and discourage-prompt Success->Fail-with-tests-removed (e.g. astropy-13236, django-11790, sympy-21612); themes summarized with GPT-5.4 help. Authors: "necessarily limited", no issue taxonomy tested.

Chen et al., "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents" · Feb 2026 · v2 (Apr 2026): §5.1.1
Turn print-probe tests into assertion-based oraclesNot measured

Recommendation from the discussion, untested. Motivating data: value-revealing prints outnumber assertions for every model (e.g. Claude Opus 4.5 25.00 prints vs 5.16 assertions per resolved task; MiniMax M2 34.06 vs 7.37); 69.7-77.5% of prints inspect raw values; assertions are mostly property (33.6-41.4%) or exact-value (34.5-42.9%) checks, relational checks 3.0-7.9%.

Chen et al., "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents" · Feb 2026 · v2 (Apr 2026): §4 Tables 4-6, §6.1, §8
Package targeted testing as a focused subagent or skillNot measured

Recommendation, untested: "the goal should not be to simply make agents write more tests, but to make testing more targeted and budget-aware ... generating one minimal regression test, strengthening a weak oracle ..., or running only the smallest relevant test slice."

Chen et al., "Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents" · Feb 2026 · v2 (Apr 2026): §6.1 Implications

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:test-first-fail-to-pass. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.

Write a fail-to-pass test before the fix, and harden it against wrong fixes