Surface information the source holds but the agent cannot read

Find information a connected source holds that the agent never reads, and expose it. At file level: a reader that drops a format (PDF, Excel, Word, ZIP) or a channel (cell colors, formatting, comments). At warehouse level: tables and columns never queried, found by comparing what the agent queried against the source's catalog and joining that to failed cases; then add them to the schema context. An index over existing metadata raised table discovery sharply. The runtime alternative, a catalog-search tool the agent calls, did not beat giving the schema up front.

Tier 3 because it changes what the model can reach and sees.

When it applies

• In the code: a file tool limited to plain-text formats; a spreadsheet reader that drops formatting; the source catalog lists tables or columns the agent's prompts and tools never reference
• In traces: failed opens of PDF, Excel, Word or ZIP files; the set of tables and columns queried, compared with the catalog, joined to failures; failure rows labeled domain-knowledge-deficit or missed-read
• In eval results: failures where the answer existed in an unread part of the source

How

1. List the source's contents (formats, catalog).
2. Diff against what the agent's tools can read and what its queries touched.
3. Join the gap to failed or insufficient-evidence cases.
4. Extend the reader, or add the missing tables to the schema context (5.b.i).
5. Paired eval on the cases that failed for lack of data.

How to measure it

• experiment (tier 3): paired eval on the questions that failed for lack of data

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

• Quality: improves, medium: Excel colors +4.9% on FACTS; Genie metadata index raised table-discovery Recall@10 from 0.075 to 0.450 (May 2026); PDF reading flat on GAIA

Caveats

The only evidence is one row of the blog's tricks table (checked against the table image, 2026-09-23). It is about background colors alone, not formatting in general, and it credits a FACTS gain to GAIA questions without saying how the two relate. Read it as one instance of 8.a.ii, not as a general spreadsheet technique.

Evidence

Evidence comes from: Data analysis, Web research, Code generation.

1. Build a knowledge graph of tables from enterprise metadata

   Table 2, internal 133-question benchmark, gpt-4o: all knowledge-graph components vs schemas only (A.1): score (% 4+) 48% vs 9%, table recall 78% vs 45%, column recall 56% vs 24%, compile 96% vs 88%. Paper: "Most of the improvement is from example queries, table clustering, and table/column attributes." Modeling-only ablation B.1 vs Full moves score only 46% -> 48%: "The knowledge graph is a better lever to improve semantic understanding."

   Albert Chen, Manas Bundele, Gaurav Ahlawat, et al. (LinkedIn), "Text-to-SQL for Enterprise Data Analytics" · Jul 2025Older evidence · §2.1, Figure 1, Table 1; §3.3.1-3.3.2, Table 2

2. Build a semantic search index from existing data assets

   Figure 4, internal table-discovery benchmark, Recall@10 with vs without semantic context by max steps: 2 steps 0.450 vs 0.075; 4 steps 0.623 vs 0.219; 8 steps 0.724 vs 0.342; 16 steps 0.774 vs 0.466. Prose: "improve table search performance by up to 40%". End-to-end (Figure 1, bundled with the rest of the Genie harness): leading code agent + Databricks MCP 32.1% at 1.00x relative cost vs Genie with Specialized Knowledge Search (Opus 4.6) 73.8% at 0.23x cost (+41.7pp).

   The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Specialized Knowledge Search section; Figure 4; Figure 1

3. Search several indices in parallel with metadata signalsNot measured

   Described only: "It uses multiple search indices in parallel together with rich metadata signals to efficiently discover most relevant assets for a user query." Not ablated separately from the index itself.

   The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Specialized Knowledge Search section

4. Put the semantic context in the prompt, not behind searchAgainstNot measured

   Reported only in passing (Section 6.5; REPORT.md 3.5): "did not significantly outperform raw-DDL Claude on this benchmark"; numbers not given in the paper. 25-table schema, so everything fit in context; the authors flag 1,000+ table warehouses as open.

   Michael Rumiantsau, Ivan Fokeev (Cube), "Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models" · Apr 2026 · Section 6.5 Limitations; REPORT.md 3.5

5. Mount raw semantic-layer files instead of retrieval toolsMixed

   Same bundled 5-query result. "The semantic layer is already great documentation ... We were building tools to summarize what was already legible. Claude just needed access to read it directly." Code: sandbox.writeFiles(files) from ./semantic-catalog.

   Andrew Qu (Vercel), "We removed 80% of our agent's tools" · Dec 2025 · v2: The file system is the agent

6. Sync terminal sessions to files the agent grepsNot measured

   "Since terminal history can be long, the agent can grep for only the relevant outputs, which is useful for logs from a long-running process like a server."

   Jediah Katz (Cursor), "Dynamic context discovery" · Jan 2026 · 5. Treating all integrated terminal sessions as files

7. Escalate from SQL to document search when data stops explainingNot measured

   Single worked trajectory (Figure 2), Phase 4 Root-Cause Investigation, steps 56-91, 18 queries + 2 doc searches: "Finds a Cost-Attribution doc"; "Escalates to docs when SQL stops explaining."

   The Databricks AI Research Team, "Pushing the Frontier for Data Agents with Genie" · May 2026 · Figure 2

8. Give the SQL agent web search for public conventionsNot measured

   Enabled on LS-Hard only (withheld groundings are web-recoverable by construction); disabled on RealEHR because groundings are institution-specific. Not ablated.

   Youngwon Lee, Jaejin Kim, Seung-won Hwang (Seoul National University), "Bootstrapping Semantic Layer from Execution for Text-to-SQL (GATE: Grounding After Test from Execution)" · Jun 2026 · Appendix B.2; §4.1

9. GAIA

   "Added PDF reading support" in the last phase, flat on train (0.24).

   Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Figure 14

10. GAIA/FACTS/SimpleQA

    Knight ships with complex-format reading; Pawn reads .txt/.md/.json/.csv only.

    Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Table 10

11. FACTS

    +4.9%; "identified that GAIA questions encode data in cell background colors".

    Scale AI, "Can AI Agents Build Better AI Agents?" · tricks table (Original template)

Related

• 5.b.iGive the model the schema it queries, in context: descriptions, keys, codes, sample values
• 8.b.iTune the search tool: snippets, enough results, retries, reformulation, filters, source quality
• 2.a.iReduce the tool surface: remove and consolidate tools, then defer the rest behind search
• 8.a.iChoose the retriever by corpus size and question type

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:surface-unreadable-information. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.