1.b.i Match the model and its effort to each call site and query class

Pick the model per role or call site rather than one model everywhere, and set its reasoning effort and sampling parameters per site; tune each role's prompt to its model (GEPA raised every model tested at lower cost). Give the agent explicit effort budgets by query class (e.g. one agent and 3-10 tool calls for simple fact-finding) so simple queries are not over-served. Upgrade the model generation before doubling token budgets. A prompt that forces stepwise reasoning is the same lever from the prompt side and hurt on GPQA; prefer the model's own effort setting.

Tier 3 because model per call site within the existing design.

When it applies

In the code: one model and one effort setting for every call site; no per-query-class budget
In traces: simple queries using as many steps, tokens or sub-agents as complex ones; a call site whose cost dominates with no quality reason; retries exhausted or timeouts at one site
In eval results: quality and cost per call site, per model; price not predicting quality across the measured configurations (in an email-triage agent we measured: Spearman 0.12)

How

From traces, list call sites and their token, cost and error profile.
For each, run a small model grid (cheaper, specialist, effort levels) on the eval slice that exercises it. Sweep effort rather than max it, since forced extra thinking cut GSM8K accuracy from 87.3% to 70.3% (1,100 to 15,980 thinking tokens), majority-voted parallel traces beat one long trace at the same budget (+10.1%), and stopping once a candidate answer appears cut reasoning tokens by 24.5% for a 2.9-point accuracy drop.
Optimize each site's prompt for its model.
Add per-query-class budgets.
Paired eval per call site.

How to measure it

experiment (tier 3): paired eval per call site and per query class

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

Quality: improves, medium: Genie: 81.4% to 91.6% with a model per role (May 2026); in an email-triage agent we measured, the best recall came from gpt-5-nano at low effort
Cost: improves, large: Genie 2.13x to 0.95x cost

Caveats

The model checkpoint itself is off-limits under VeRO's rules; only its call parameters move.

Evidence

Evidence comes from: Data analysis, Code generation, Reasoning, Web research, Tool use.

Assign a different LLM to each sub-agent role

Figure 1, internal benchmark: + Multi-LLM (GPT-5.4 + Opus 4.6) on top of parallel thinking 81.4% -> 91.6% (+10.2pp) while relative cost falls 2.13x -> 0.95x (baseline code agent = 1.00x). Bundled with GEPA-optimized prompts ("Multi-LLM design with optimized prompts").

The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Multi-LLM section; Figure 1; Figure 3
Optimize each sub-agent prompt per model with GEPA

Figure 6, end-to-end accuracy of the Multi-LLM agent vs relative token cost (Sonnet 4.5 baseline = 1.0), approximate reads, baseline -> GEPA-optimized table-search sub-agent: Sonnet 4.5 ~73.1% @1.00 -> ~78.2% @~0.80; GPT-5.4 ~71.8% @~0.76 -> ~78.2% @~0.74; GPT-5.4-mini ~75.3% @~0.72 -> ~76.9% @~0.66; Opus 4.6 ~66.7% @~0.98 -> ~74.4% @0.90. Every model gained accuracy and lost cost; weakest baseline (Opus 4.6 as table-search sub-agent) gained most.

The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Multi-LLM section; Figure 6
Dispatch distinct base prompts per model familyNot measured

OpenCode "nine base prompts by model-id substring"; Codex per-generation prompts delivered as server-side catalog data; Hermes "15 guidance blocks gated by model family (tool-use enforcement blocks are sent to GPT/Codex/Gemini/Grok/Qwen/DeepSeek models and never to Claude, with code comments citing observed per-model failures)."

Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Sec 7.2; Observation 3; Table 12 "Model-Family Prompt Matrix"
Keep a per-model metadata registryNot measured

Aider "per-model metadata for 350+ models"; "This data-driven approach enables support for new models without code changes"; Pi "20 per-model compat quirk flags"; OpenCode 1,400-line per-vendor transform matrix.

Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Sec 7.1; Sec 12.3; Recommendation 2
Select edit format / edit toolset per modelNot measured

Aider picks among 13 edit formats via Coder.create() from model metadata; OpenCode "GPT-family models receive a port of Codex's apply_patch DSL and lose edit/write entirely"; OpenHands "model-specific tool presets (default, gemini, gpt5, planning)". The paper calls editing strategy "one of the most important determinants of code-modification accuracy" (unmeasured).

Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Sec 8.4; Observation 4; Sec 15.4; Table 11 "Polymorphic Edits"
Upgrade the model generation before doubling token budget

"upgrading to Claude Sonnet 4 is a larger performance gain than doubling the token budget on Claude Sonnet 3.7" (BrowseComp analysis; no absolute numbers).

Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford (Anthropic), "How we built our multi-agent research system" · Jun 2025Older evidence · Benefits of a multi-agent system
Use a non-reasoning model as the tool-calling policyNot measured

"The reasoning LLM (DeepSeek-R1) is not chosen for AutoLink's policy due to observed instruction following degradation ... which occasionally leads to deviations from the required thought and action format." Not quantified.

Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, Xiang Bai (HUST, Fudan, Xiaomi), "AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale" · Nov 2025 · Implementation Details, Experimental Setup
GPQA

Train 0.61 to 0.58 (regressed) in phase 1.

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 13
GAIA/FACTS/SimpleQA (Knight agent)

Knight's 140-line prompt bakes in CoT, ReAct and self-verification; it is harder to improve but far more stable than Pawn.

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper §4.3, Table 10
five follow-up benchmarks

8 to 14 of 20 runs per benchmark.

Scale AI, VeRO code release · figure_01
Embed effort-scaling rules keyed to query complexityNot measured

"Simple fact-finding requires just 1 agent with 3-10 tool calls, direct comparisons might need 2-4 subagents with 10-15 calls each, and complex research might use more than 10 subagents". Fixes errors "like spawning 50 subagents for simple queries"; over-investment "was a common failure mode in our early versions."

Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford (Anthropic), "How we built our multi-agent research system" · Jun 2025Older evidence · Prompt engineering: Scale effort to query complexity
Skip reflection on easy questions and prefetch metadataNot measured

Stated for speed only: "it evaluates question difficulty, bypasses self-reflection for simple queries, and pre-fetches table metadata to minimize routine tool calls." Not quantified.

Albert Chen, Manas Bundele, Gaurav Ahlawat, et al. (LinkedIn), "Text-to-SQL for Enterprise Data Analytics" · Jul 2025Older evidence · §2.3.1
Scale retrieval depth to query difficultyNot measured

The rationale is to control 'online access cost through query-adaptive retrieval'. No numbers are quoted.

Xiaofang Yang, Lijun Li, Heng Zhou, Tong Zhu, Xiaoye Qu, Yuchen Fan, et al. (Shanghai AI Lab et al.), "Toward Efficient Agents: A Survey of Memory, Tool Use, and Planning" · Jan 2026 · §3.3.1 Rule-enhanced Retrieval; Hierarchical Retrieval
Call tools only when model confidence is lowNot measured

Xu et al.'s method 'could reduce the number of tool calls'. Framing: 'an efficient tool user... calls tools only when the expected reduction in uncertainty or error justifies the cost'.

Xiaofang Yang, Lijun Li, Heng Zhou, Tong Zhu, Xiaoye Qu, Yuchen Fan, et al. (Shanghai AI Lab et al.), "Toward Efficient Agents: A Survey of Memory, Tool Use, and Planning" · Jan 2026 · §4.2 Cost-Aware Tool Calling; §4.3 Selective Invocation; §4.4 'Knowing When Not to Call Tools'
Pick respond/retrieve/tool/verify/stop by estimated utilityNot measured

Described only qualitatively. The survey's point is that efficient tool calling must control 'when to stop, verify, or search'.

Xiaofang Yang, Lijun Li, Heng Zhou, Tong Zhu, Xiaoye Qu, Yuchen Fan, et al. (Shanghai AI Lab et al.), "Toward Efficient Agents: A Survey of Memory, Tool Use, and Planning" · Jan 2026 · §4.2 Efficient Test-Time Scaling
Default to a fast policy, escalate to slow planningNot measured

SwiftSage reports token use and time steps, but the survey quotes no numbers. The rationale is 'reserving expensive reasoning for difficult decisions'.

Xiaofang Yang, Lijun Li, Heng Zhou, Tong Zhu, Xiaoye Qu, Yuchen Fan, et al. (Shanghai AI Lab et al.), "Toward Efficient Agents: A Survey of Memory, Tool Use, and Planning" · Jan 2026 · §5.1 Adaptive Budgeting and Control; §7 (MLLM fast-slow)
Adds a "Think step by step" prompt, against an answer-directly prompt and a no-instruction default, on reasoning and non-reasoning models, 25 trials per question

Reasoning models, average rating against direct: o3-mini RD = 0.029 (p = .024), o4-mini RD = 0.031 (p = .003), Gemini 2.5 Flash RD = -0.033 (p = .005), and Flash 2.5 lost 0.131 on the 25-of-25-correct metric; "CoT requests took between 20-80% (10-20 seconds) longer than direct requests." Non-reasoning models gained against direct (Gemini 2.0 Flash RD = 0.135, Sonnet 3.5 RD = 0.117); against the no-instruction default, Gemini 2.0 Flash still gained significantly (RD = 0.062, p < .001) while Sonnet 3.5 did not (RD = -0.019, p = 0.189). "For dedicated reasoning models, the added benefits of explicit CoT prompting appear negligible and may not justify the substantial increase in processing time."

Lennart Meincke, Ethan Mollick, Lilach Mollick, Dan Shapiro (Wharton Generative AI Labs), "Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting" · Jun 2025Older evidence · Summary; Results, Figures 1-3; Figure S2; Discussion and Conclusion
Reasoning models made to keep re-checking by suppressing the end-of-thinking token and appending "Wait", against parallel thinking: the same token budget spread over independent traces with a majority vote

DeepSeek-R1-Distill-Qwen-1.5B on GSM8K: "accuracy increases from 82.2% to 87.3% as the average number of thinking tokens increases from 385 to 1100", then "pushing the average thinking token count from 1100 to 15980 reduces accuracy from 87.3% to 70.3%". At a 16,000-token budget, "Wait & Think More" gave "a 11.8% performance drop" against standard thinking, while parallel thinking with the same budget "boosts accuracy by 10.1%". The authors report "a consistent non-monotonic trend in performance across multiple tasks and datasets".

Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, Amrit Singh Bedi, "Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models" · Jun 2025Older evidence · §2.3 Figure 2; §4 Figures 6-7
Stops reasoning early once candidate answers appear, after finding most later reflections only confirm the first answer

"over 90% of the reflections are confirmatory"; corrective ones (wrong to right) are "mostly less than 2%". Reflections after the first candidate took 16.8% to 47.8% of the tokens for a gain of 1.4% to 3.5%. Stopping early on Qwen3-8B cut reasoning tokens by 24.5% for a 2.9-point average accuracy drop (84.7 to 81.8).

Liwei Kang, Yue Deng, Yao Xiao, Zhanfeng Mo, Wee Sun Lee, Lidong Bing, "First Try Matters: Revisiting the Role of Reflection in Reasoning Models" · Oct 2025Older evidence · §2.2 Figures 3-4; §4 Table 2

Also known as

Chain-of-thought
Routing to specialists

9.c.iiSpend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays
6.b.iiiUse multi-agent orchestration only for breadth-first, independent sub-tasks
6.c.iCode-then-execute (CaMeL)
4.c.iEpisodic memory retrieval
7.c.ivGraph-of-Thought
7.d.iiInversion of control
5.c.iiSub-agent architectures with clean contexts
7.c.vLanguage Agent Tree Search (LATS)
6.b.iiLLM map-reduce
7.c.viMulti-agent debate
6.a.iOracle-worker multi-model
6.c.iiPlan-then-execute
7.c.iReflection loop
3.c.iSelf-Discover reasoning structures
7.c.viiTree-of-Thought

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:model-and-effort-per-call-site. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.

Match the model and its effort to each call site and query class