2.b.ii Tell the agent when to use each tool, in context it always sees

For each tool or skill that is under-used or misused, add an explicit trigger rule where the model always sees it (system prompt, AGENTS.md): the situation that calls for the tool, and what to use it instead of (e.g. the dedicated read or edit tool rather than bash). Test the wording, since results swing with it. It will not make a model that rarely calls tools start calling them. The cheapest form is one sentence per tool stating what it is for: that alone recovered most of a 34.7-point loss caused by offering the model an irrelevant tool.

Tier 3 because it changes what the model sees.

When it applies

In the code: tools or skills whose intended use is stated nowhere the model always sees; overlapping tools with no rule for choosing between them
In traces: tool spans: a tool called on under about 5% of cases where its trigger condition held (e.g. a docs skill never invoked in 56% of runs); bash spans where a dedicated tool exists; failure rows labeled incorrect-tool-selection
In eval results: failures where the right tool existed but was not called

How

From traces, list tools with a low call rate on cases where they would have helped, and tools chosen wrongly.
For each, write one rule: when to use it, and instead of what.
Put the rules in always-loaded context, not in an optional skill.
A/B two wordings; keep the one with the higher call rate and pass rate.

How to measure it

experiment (tier 3): paired eval with and without the rule, plus the tool's call rate in traces

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

Quality: improves, large: Vercel: pass rate 53% to 79% once told when to invoke the docs skill (Jan 2026); VeRO mixed: -5 and +4.5 points; one sentence of tool scope recovered most of a 34.7-point loss from an irrelevant tool (Paturi et al., Sep 2026)

Evidence

Evidence comes from: Reasoning, Web research, Code generation, Tool use.

MATH

Phase 3, 'Enhanced tool usage instructions', added right after phase 2 added a SymPy calculator tool: validation 0.917 to 0.867 (regressed), train flat at 0.813. The figure gives only the label, not the instruction text.

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 12 (Orchestrator, Sonnet 4.5; 'Early Peaker')
SimpleQA

Phase 1, 'Improved prompt, tool descriptions', bundled with prompt changes: validation 0.577 to 0.622, train 0.456 to 0.587. Not separable from the description changes (T35).

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 15 (Orchestrator, Opus 4.5; 'Best SimpleQA')
Tell the agent in AGENTS.md when to invoke a skill

Trigger rate to 95%+, pass rate 53% -> 79% (+26pp); Build/Lint/Test 95/100/84. Still 21pp below the AGENTS.md index. Vercel calls the wording sensitivity "brittle for production use".

Jude Gao (Vercel), "AGENTS.md outperforms skills in our agent evals" · Jan 2026 · Explicit instructions helped, but wording was fragile
Do not rely on the agent choosing to invoke a skill

Skill never invoked in 56% of eval cases; pass rate 53% = baseline 53% (+0pp); on tests 58% vs baseline 63%, lint 89% vs 95%.

Jude Gao (Vercel), "AGENTS.md outperforms skills in our agent evals" · Jan 2026 · Skills weren't being triggered reliably
Prefer dedicated tools over shell in instructionsNot measured

Claude Code "Do NOT use the Bash tool to run commands when a relevant dedicated tool is provided ... allows the user to better understand and review your work."

Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Sec 7.3 "Tool-use philosophy"
Generate tool guidance from the active toolsetNot measured

Pi "each tool contributes promptSnippet/promptGuidelines, so the prompt co-varies with the active toolset"; whole prompt is 30–40 lines.

Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Sec 7.2 (Pi)
Give explicit tool-selection heuristics in the promptNot measured

"an agent searching the web for context that only exists in Slack is doomed from the start" ... "Bad tool descriptions can send agents down completely wrong paths, so each tool needs a distinct purpose and a clear description."

Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford (Anthropic), "How we built our multi-agent research system" · Jun 2025Older evidence · Prompt engineering: Tool design and selection are critical
Prompting weak models to search does not make them searchAgainst

Same prompt explicitly telling the model to use the tool: Qwen3-32B 0.89-0.94 search calls/q, Search-R1-32B 1.68-1.78, vs gpt-5 / o3 >20. With oracle evidence in context Qwen3-32B reaches 83.25% (gpt-4.1 93.49%), vs 10.36% when it must search.

Chen et al., "BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent" · Aug 2025Older evidence · Sec 4.3; Sec 4.6; Sec 4.8.1
State in one sentence what each tool is for

Same setup: a one-sentence instruction stating the tool's scope recovers most of the lost answers.

Paturi et al., "When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering" · Sep 2026

Also known as

Tool docstrings with use / do-not-use guidance and examples

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:tool-use-guidance. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.

Tell the agent when to use each tool, in context it always sees