Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit

State the required answer format in the prompt, and make the harness extract it robustly: strip markers, normalize numbers, detect refusals, and force a best-effort answer when the loop ends on a budget rather than returning nothing. The format reminder alone was mixed on GPQA (train up, test down); the harness side is what stops right work being lost.

Tier 2 because graceful degradation: always return a usable answer.

When it applies

• In the code: no answer-format statement; extraction that assumes a marker; no forced answer when the loop ends on a budget
• In traces: runs ending with no extractable answer; refusals returned as answers; numbers in the wrong format; runs ending at the turn cap with a correct value in context (in one image-labeling agent, 71 items returned all-unknown)
• In eval results: failures where the working was right but the submitted answer was not (instruction-grader-mismatch)

How

1. From traces, list runs whose working was right but whose returned answer was missing, malformed or a refusal.
2. Add the format section to the prompt (3.a.iv).
3. Harden extraction and add the forced-answer path at the turn cap.
4. Measure no-answer and malformed-answer rates.

How to measure it

• before after (tier 2): no-answer, malformed-answer and refusal rates on the same cases

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: improves, medium: estimate; in an image-labeling agent we measured, a salvage turn at the limit cut all-unknown items from 71 to 1
• Cost: none
• Performance: none

Evidence

Evidence comes from: Code generation, Reasoning, Web research.

1. five follow-up benchmarks

   Symbols such as _submit, _answer_payload, _scale_variants, _format_number, _extract_answer are common enough to get deterministic role hints.

   Scale AI, VeRO code release · taxonomy.py role hints

2. SWE-Atlas-QnA

   15/20 runs touched submission.

   Scale AI, VeRO code release · figure_01

3. GPQA

   Train recovered 0.54 to 0.60 in phase 2; test 0.54 (below the 0.61 baseline).

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 13

4. SWE-Atlas-QnA, GAIA-Shell

   Submission-path edits in 15/20 and 18/20 runs; 2/20 on Terminal-Bench, where the verifier does not read an answer string.

   Scale AI, VeRO code release · figure_01

Related

• 3.a.ivRestructure the system prompt: every requirement once, up front, in named sections; nothing the model doesn't need
• 9.b.iiiVerify completion against the environment before the agent stops
• 7.b.iiKeep errors in context
• 2.b.iMake tool calls and tool results fail soft, and retry transient failures
• 3.a.iAdd worked examples to the prompt (few-shot), or curated example queries for data agents
• 7.c.viMulti-agent debate
• 3.c.iSelf-Discover reasoning structures
• 7.b.iiiSize run budgets so productive runs aren't cut off: turn caps, deadlines, command timeouts

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:extractable-final-answer. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.