Tune the search tool: snippets, enough results, retries, reformulation, filters, source quality

Make the search tool return what the model needs to choose what to open (a snippet per result, not just titles; five or more results; a full-document reader beside truncated search), retry failed searches, and reformulate when a query returns little (start broad, then narrow; use the corpus's own terms) rather than retrying it verbatim. Expose the source's metadata as filter parameters (date, product, version, source) where the corpus has it. Drop results from sources that leak the answer or are untrustworthy. Expanding one question into many queries up front has not helped.

Tier 2 because retries recover failed searches.

When it applies

• In the code: a search tool returning titles or URLs only; a fixed small result count; no retry; no filter parameters over a corpus with metadata; results passed to the model unfiltered
• In traces: many page fetches per question; queries returning zero results followed by the same query or by giving up; results from the wrong product, version or date; failure rows labeled missed-read
• In eval results: retrieval recall below answer accuracy needs; failures where the answer was in an unfetched result

How

1. From traces, count fetches per question, empty-result queries, and repeats of the same query.
2. Change one parameter at a time (snippets, result count, retries, filters).
3. Paired eval scored on retrieval recall and answer accuracy.

How to measure it

• before after (tier 2): failed-search rate
• experiment (tier 3): paired eval on retrieval recall and answer accuracy

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: mixed, medium: snippets: SimpleQA train +0.15 but validation -0.06; source filtering +3.3% on FACTS; retries and result count unmeasured; up-front query expansion: no gain (LinkedIn, 2025)
• Cost: mixed, small

Evidence

Evidence comes from: Web research, Data analysis.

1. BrowseComp-Plus

   9/20 runs, and almost none elsewhere; the only benchmark with a retrieval corpus.

   Scale AI, VeRO code release · figure_01

2. SimpleQA

   Train 0.74 to 0.89, validation 0.80 (down from 0.86 best).

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 15

3. FACTS

   +3.3%.

   Scale AI, "Can AI Agents Build Better AI Agents?" · tricks table (Tool-Centric template)

4. GAIA/FACTS/SimpleQA

   Knight "Enhanced (5+ results, retry logic)" vs Pawn "Basic (3 results, no retries)".

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Table 10

5. Decompose the question with a query plannerAgainst

   Appendix A.5 negative result, gpt-4o: "the query planner lowered recall and quality metrics"; produced overly nested queries even after instructing it to minimize tasks with few-shot examples. Hypothesis: unnecessary for gpt-4o, possibly useful for smaller models.

   Albert Chen, Manas Bundele, Gaurav Ahlawat, et al. (LinkedIn), "Text-to-SQL for Enterprise Data Analytics" · Jul 2025Older evidence · Appendix A.5

6. Use planner sub-tasks as retrieval query expansionAgainst

   Appendix A.5: "this did not improve the recall of tables retrieved from EBR." Authors suggest an explicit query-expansion prompt might do better (untested).

   Albert Chen, Manas Bundele, Gaurav Ahlawat, et al. (LinkedIn), "Text-to-SQL for Enterprise Data Analytics" · Jul 2025Older evidence · Appendix A.5

7. Prompt searches to start broad, then narrowNot measured

   "Agents often default to overly long, specific queries that return few results. We counteracted this tendency by prompting agents to start with short, broad queries, evaluate what's available, then progressively narrow focus."

   Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, Daniel Ford (Anthropic), "How we built our multi-agent research system" · Jun 2025Older evidence · Prompt engineering: Start wide, then narrow down

8. Let the agent write its own targeted schema-retrieval queries

   Table 4: removing @retrieve_schema costs 6.7-7.6 SRR points (see index entry). Table 7, top-m per call: m=1 SRR 89.2 (20.7K tokens), m=2 89.6 (21.0K), m=3 91.2 (21.2K). Contrasted with CHESS-style question rewriting; not directly compared.

   Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, Xiang Bai (HUST, Fudan, Xiaomi), "AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale" · Nov 2025 · Action Space @retrieve_schema; Appendix Details of Action Space; Table 4; Table 7

Related

• 8.a.iChoose the retriever by corpus size and question type
• 8.a.iiSurface information the source holds but the agent cannot read
• 5.b.iGive the model the schema it queries, in context: descriptions, keys, codes, sample values
• 9.a.iiVerify citations claim by claim for factual support
• 8.b.iiLink large schemas by exploration: start broad, grow the linked set with probing queries, favor recall

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:tune-search-tool. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.