Choose how the agent acts: typed tools, code execution, or shell, matched to the model and task

Decide the action interface deliberately, by model and by task, and provision what it needs. The rule the evidence supports: (1) weaker models get typed tools (read, write, edit, grep, a narrow calculator); a general code interpreter made a weak agent write boilerplate and fail until it was replaced with typed functions. (2) Structured data gets a typed query tool, whatever the model: SQL scored 100% against 52.7% for bash at a seventh of the tokens, and SQL plus bash also reached 100%. (3) Strong models on shell-heavy or general office work get bash: on enterprise digital-worker tasks bash-only beat typed tools by 21.8-24.5 points with fewer tokens, and typed tools on top of bash added nothing. (4) Work that chains many calls or processes bulk data gets code execution, because intermediate results never pass through the model; calling tools from code matched or beat JSON calling for 11 of 14 models on BFCL v4, and a narrow calculator did as well as full code execution on maths. Whichever you choose: start minimal and add tools per observed failure; for code or shell, provision the packages and fixtures the agent expects, allow concurrent commands, and sandbox it (10.a.i).

Tier 3 because it changes the agent's tools within the existing design.

When it applies

• In the code: the interface in use (typed tools, code interpreter, bash), the model it serves, and whether the environment has the packages and fixtures the tasks need
• In traces: bash-only runs emitting tools that do not exist; typed-tool runs re-patching the same file; code errors in routine operations (reading a file, searching); import-error or command-not-found spans at the start of runs; calculations done in the model's own text; independent commands serialized
• In eval results: success and cost by model under each interface

How

1. Classify the model's shell and coding ability from a probe run (typed tools against bash-only on 20 cases).
2. Pick the interface by the table in the summary; for code execution, install the packages the tasks need and set up fixtures.
3. Add tools only when a trace shows a failure a tool would have prevented.
4. Paired eval per model: success, cost, steps.

How to measure it

• experiment (tier 3): paired eval per model, interface A against B: success, cost, steps

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: mixed, large: typed tools +15.0 points for a weak model; bash-only +3.6 for a strong one (Fan, Sep 2026); typed functions in place of an interpreter took GAIA from 0.07 to 0.24 (VeRO)
• Cost: improves, large: bash-only -53% cost for a bash-capable model; chaining tools in code 150k to 2k tokens in one example

Caveats

The paper flags this as intuitively helpful but the test gain is within noise; MATH baseline 0.87 is near the target model's ceiling.

Evidence

Evidence comes from: Code generation, Tool use, Data analysis, Web research, Reasoning, Retrieval.

1. Match action space to model's shell proficiency and task

   T4/128k, planning on. Nemotron-3 30B: typed tools +15.0 pts SWE (25.2 vs 10.2) and +10.1 TB. 66% of its bash-only TB runs end after emitting tools that don't exist (trajectory 71->15 turns). 120B: tools +1.6 SWE / +4.5 TB and shorter runs (101->77, 96->70 turns) at no extra cost. 550B: bash-only +3.6 SWE / +5.6 TB with cost -53% / -30% and calls -32% / -24%. Mistral: tools +23.2 on SWE (bash-only 45.4 vs 68.6; 32.8% of bash-only runs never edit vs 1.2%; file-localization failures 16.0%->41.4%), but bash-only +6.7 on TB, where 71.9% of its actions were already bash. Bash-only cuts re-patching for all models (e.g. 550B 4.6->1.5) and raises create/replace share on TB (e.g. 28%->64% for 30B). The ablation bundles tool availability, prompts, file-state tracking and diagnostics.

   Fan et al., "An Empirical Study of Harness Design for Coding Agents" · Sep 2026 · §2.2 Table 1; §3.2 'The predefined tool set scaffolds weaker models...', Fig 7; §4 Table 7; App 9.3.1 Table 10, 9.5 Tables 11-12

2. Replace specialized text-to-SQL tools with bash plus SQL

   Claude Opus 4.5, 5 representative queries, old vs new: avg time 274.8s -> 77.4s ("3.5x faster"); success 4/5 -> 5/5; avg tokens ~102k -> ~61k ("37% fewer"); avg steps ~12 -> ~7 ("42% fewer"). Worst case: old "724 seconds, 100 steps, and 145,463 tokens before failing"; new "141 seconds with 19 steps and 67,483 tokens" and succeeded. v1 code lists 17 tools. Bundled with the context and prompt removals below, so not a single-variable result.

   Andrew Qu (Vercel), "We removed 80% of our agent's tools" · Dec 2025 · v2: The file system is the agent; 3.5x faster table

3. Drop reasoning-constraining prompts and pre-filtered context

   Part of the bundled v1 -> v2 change (5 queries, Opus 4.5, numbers above). "We were constraining reasoning because we didn't trust the model to reason. With Opus 4.5, that constraint became a liability." Both code samples show instructions: "", so the size of the prompt change is not visible.

   Andrew Qu (Vercel), "We removed 80% of our agent's tools" · Dec 2025 · Getting out of the model's way; Lessons learned

4. Start with a bash-only tool, add tools per failure

   "Mini-SWE-Agent's single-tool design reports 74%+ on SWE-Bench with essentially no tool infrastructure (self-reported)"; the paper says it is not comparable head-to-head with Codex's 69.1% (different models, runs, sandboxing).

   Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Recommendation 3 (Sec 16.3); Sec 13.4; Sec 15.2

5. Consolidate APIs instead of 1-to-1 wrapper toolsNot measured

   Rec 17 quoting [18]: "a common error we've observed is tools that merely wrap existing software functionality or API endpoints." Consolidate instead.

   Barbaste et al., "Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems" · Jul 2026 · Recommendation 17 (Sec 16.9)

6. Give the agent a filesystem and shellNot measured

   "The filesystem gives agents access to persistent context. The shell lets agents run built-in utilities, CLIs, provided scripts, or code they write." Examples: Claude Code, Manus virtual computer.

   Lance Martin, "Agent Design Patterns" · Jan 2026 · Give Agents A Computer

7. Keep few atomic tools; push actions into bashNot measured

   Rationale: the GitHub MCP server "has 35 tools with ~26K tokens of tool definitions"; overlapping tools confuse models. Claude Code uses "around a dozen tools", Manus "fewer than 20"; Manus action space is "a hierarchy".

   Lance Martin, "Agent Design Patterns" · Jan 2026 · Multi-Layer Action Space

8. Give a read plus grep tool pair as minimum retrieval surface

   BrowseComp-Plus 100-q subset, GPT-5.4 nano, L4 context policy: read+grep 61% acc, 19 tool calls, $0.0355/q vs Qwen3-Embedding-8B retrieval 45%, 18 calls, $0.0498/q vs BM25 32%, 19 calls, $0.0527/q.

   Li et al., "Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction" · May 2026 · Sec 4.3 RQ6; Table 5

9. Allow piped shell commands for chained lexical search

   BrowseComp-Plus 100-q subset, GPT-5.4 nano: open bash 73% vs read+grep 61% (+12), but 35 vs 19 tool calls and $0.1021 vs $0.0355 per q. Lite bash usage (3,168 commands): rg | head 56.2%, rg | rg 20.6%, wc 7.8%, single rg 6.3%, ls 4.3%, python -c 2.4%, cat 0.1%. CC: Bash 62.4% / Grep 33.0% of calls; within Bash, chained search 22.3%, local context peek 18.0%, regex 17.0%, file localization 14.0%, full-document read 9.1%.

   Li et al., "Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction" · May 2026 · Sec 4.3 RQ6, Table 5; Appendix B.1, Fig 6; Fig 4 (right)

10. MATH

    Validation 0.88 to 0.92 (best of run) and test 0.86 to 0.88 with T07; train stayed at 0.81.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 12

11. MATH

    Added in phase 5 (train 0.90, validation 0.90) then removed in phase 6; no lasting gain over T08.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 12

12. GAIA

    The seed's CodeInterpreter tool was replaced with plain function tools (see T15) and that was the biggest gain of the run.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 14 (Default, Sonnet 4.5)

13. GAIA

    Validation 0.07 to 0.24 in one phase; the largest single jump in any example trajectory. Final 0.30, the best GAIA result in Table 2.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 14

14. GAIA/FACTS/SimpleQA

    Knight 12+ packages vs Pawn 3 (math, datetime, json, re).

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Table 10

15. Terminal-Bench and others

    MAX_CONCURRENT_SHELLS raised twice, never lowered.

    Scale AI, VeRO code release · figure_04

16. five follow-up benchmarks

    3 to 13 of 20 runs per benchmark (13 on GAIA-Shell, whose seed is empty).

    Scale AI, VeRO code release · figure_01

17. Give a bash-only interface rather than typed tools for enterprise digital-worker tasks

    Opus-4.8 and GPT-5.5 on TheAgentCompany and APEX-Agents, five interfaces compared: bash-only beats typed tools by 21.8-24.5 pp on TheAgentCompany and 4.8-7.4 pp on APEX-Agents, with 19-72% fewer tokens. Adding typed or agent-synthesized tools on top of bash gives no measurable gain; programmatic tool calling saves tokens but scores below bash.

    Mak et al., "Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents" · Sep 2026

18. Call tools from code rather than JSON

    BFCL v4, 14 models: programmatic (code) tool calling matches or beats JSON tool calling in 11; more robust under degraded context, where the JSON baseline loses 2.3% on average.

    Patel et al., "The Bitter Lesson of Tool Calling" · Aug 2026

19. Call tools programmatically from code

    Programmatic tool calling cut tokens 37% (43.6k to 27.3k) on complex research tasks.

    Bin Wu et al. (Anthropic), "Introducing advanced tool use on the Claude Developer Platform" · Nov 2025

20. Expose a large API through search and execute in codeNot measured

    2,500+ API endpoints behind search() and execute(); the model writes code against the typed spec in a sandbox. About 1k tokens against 1.17M, calculated not tested.

    Matt Carey (Cloudflare), "Code Mode: give agents an entire API in 1,000 tokens" · Feb 2026

21. Use code mode for multi-tool tasks onlyMixedNot measured

    About 30% fewer tokens on identical tasks (one author's test, Opus 4.5); little benefit on single-tool tasks; model-dependent. A description mentioning 'long-running task' pulled in the wrong extension; explicit imports prevented it.

    Rizel Scarlett (Block), "8 Things You Didn't Know About Code Mode" · Feb 2026

22. Give a typed query tool for structured data, not bash aloneMixed

    Structured-data questions, Sonnet 4.5: SQL tool 100% accurate, bash 52.7%, basic file tools 63%; bash used 7x the tokens and cost 6.5x more. SQL plus bash reached 100% with the agent checking its own answers, at about 2x SQL's tokens. Found and fixed 5 wrong expected answers in the benchmark.

    Ankur Goyal (Braintrust) and Andrew Qu (Vercel), "Testing if 'bash is all you need'" · Jan 2026

Also known as

• CodeAct

Related

• 2.a.iReduce the tool surface: remove and consolidate tools, then defer the rest behind search
• 6.c.iCode-then-execute (CaMeL)
• 10.a.iConstrain what the agent can touch
• 9.a.iFeed back a sound external verifier's verdict; never the model's own critique
• 2.c.iBound tool output: cap it, or write it to a file the agent can read
• 3.b.iiMask tools instead of removing them
• 6.b.iParallel tool execution for read-only calls
• 7.d.iiiReAct loop
• 4.c.iiSkill library evolution

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:choose-action-interface. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.