BIRD

Fits single-shot text-to-SQL agents that answer questions over one database with a known schema. Borrow its execution-accuracy grading, the evidence hint pattern and the human baseline (92.96 execution accuracy) as a reference point.

What it asks the agent to do

• translate a natural-language question into one SQLite query
• use the given external knowledge hint to map business terms to columns
• handle dirty values and large database contents when writing SQL
• write SQL that runs efficiently on a large database

What to borrow from it

baselinegrading designcase template

Caveats

Test set is hidden and scored by the maintainers; the dev set has likely leaked into model training data; SQLite only, one query per question, no multi-turn; the top entry (82.95) is within ten points of the human score.

Details

Links

• Paper
• Github
• Leaderboard