Restructure the system prompt: every requirement once, up front, in named sections; nothing the model doesn't need

Rewrite the system prompt so that each requirement the task has (constraints, the answer format, the completion check, how to work efficiently, the domain procedure) is stated once, up front, in a named section, and remove instructions the model does not need. Adding structure and removing bulk are the same edit seen from two sides: what stays is what the eval shows the model needs. A domain procedure section is worth adding only when failures cluster on a step an expert would take; write it from those failures, from a domain expert, or by searching prompt variants against the eval (GEPA).

Tier 3 because it changes what the model sees.

When it applies

• In the code: system prompt with no named sections; requirements repeated, contradictory, or absent (no answer format, no completion criteria); prompt plus tool descriptions over about a quarter of the context window
• In traces: failure rows labeled instruction-following-failure or instruction-grader-mismatch where the requirement was never in the prompt; prompt tokens the largest share of input per step; context-overflow outcomes
• In eval results: an ablation that removes a section and shows no loss (cut it), or failures that cluster on one step of the task procedure (add it)

How

1. List the requirements the task actually has, from the task spec and from failures.
2. Put each in one named section; put the answer format and completion check in their own sections.
3. Cut everything else, and shorten tool descriptions and checklists with it (the TerminalBench run cut the prompt from 7 lines to 3).
4. Add a domain procedure only if failures cluster on a step; derive it from those failures or an expert, or generate and test variants with GEPA.
5. Run the paired eval on development and held-out cases; keep only sections that earn their place.

How to measure it

• experiment (tier 3): paired eval, old against restructured prompt, checked on held-out cases; count context-overflow failures too

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: mixed, small: adds: MAST +4.7 to +5.0 points on GSM-Plus (2025), VeRO MATH validation +0.10 with train -0.07; cuts: TerminalBench +6.8 points where the prompt crowded out context, GPQA train -0.04 where the prompt carried procedure
• Cost: improves, small: fewer prompt tokens
• Performance: improves, small

Caveats

Intermediate 20-task subset showed 45% and misled the optimizer; full evaluation regressed previously-passing tasks. 12 new passes against 6 regressions; the same run left 11 AttributeError crashes in place. Helps when the prompt was crowding out task context; hurts on reasoning tasks where the prompt carried procedure.

Evidence

Evidence comes from: Code generation, Reasoning.

1. TerminalBench-2

   8 lines to 23 lines in Filesystem-B=178. Net-zero pass rate (6 new passes, 6 regressions) although the run had the largest crash reduction (46.1% to 30.3%).

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper §A.4

2. Move self-correction criteria into the initial prompt

   Table 8, CommonGen-Hard concept coverage, gpt-3.5-turbo-0613: Madaan et al. reported standard 44.0 -> self-correct 67.0 (7 calls); replication 53.0 -> 61.1; adding "Write a reasonable paragraph that includes *ALL* of the above concepts" gives 81.8 in 1 call, and self-correcting that drops it to 75.1.

   Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" · Oct 2023Older evidence · Sec. 5; Table 8

3. Put every task requirement in the initial promptNot measured

   Self-Refine used wrong few-shot labels or mismatched instructions only for initial generation (Tables 9-10); detoxification studies often do not ask for harmless output initially.

   Kamoi et al., "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs" · Jun 2024Older evidence · Sec. 4 Weak Initial Responses; Table 4

4. Restructure prompt with a dedicated verification section

   Table 5, GSM-Plus (200 sampled problems, 6 repetitions): GPT-4 84.75±1.94 -> 89.75±1.44; GPT-4o 84.25±1.86 -> 89.00±1.38 (Wilcoxon p=0.03 vs baseline for both).

   Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" · Mar 2025Older evidence · App. H.1, App. L.1-L.2, Table 5

5. Specify each agent's role and task clearlyNot measured

   Proposed tactic (App. L.2 is the example; measured forms are the ChatDev/AG2 entries above). Such tactics "often prove inconsistent".

   Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" · Mar 2025Older evidence · App. G.1, Table 4

6. TerminalBench-2

   Five-line block in Tools-B=89 alongside T01 and T02; that commit went 30.3% to 33.7% pass rate.

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper §A.4

7. TerminalBench-2

   "Tools-B=178: prompt 7 to 3 lines, tool descriptions to one phrase each, checklist 10 to 4 lines, cap 17KB. Best pass rate of the study, 37.1% (+6.8pp), with ContextLengthExceeded 13 to 2, without fixing the parser crash."

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper §A.4, Table 16

8. MATH

   Validation 0.78 to 0.88 in phase 1; train dropped 0.85 to 0.78 in the same phase.

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 12 (Orchestrator, Sonnet 4.5)

9. MATH

   Flat on validation (0.87).

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 12

10. GPQA

    Train 0.58 to 0.54 (regressed).

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 13 (Resources Only, Sonnet 4.5)

11. TerminalBench-2

    As part of T06, the best result in that study.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper §A.4

Also known as

• System prompt altitude and sectioned structure

Related

• 3.a.iiAvoid few-shot ruts
• 2.b.iiTell the agent when to use each tool, in context it always sees
• 9.b.iMake the final answer reliably extractable: state the format, extract robustly, force an answer at the limit
• 3.b.iKeep the prompt prefix cache-stable
• 3.a.iAdd worked examples to the prompt (few-shot), or curated example queries for data agents
• 3.c.iSelf-Discover reasoning structures

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:restructure-system-prompt. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.