Mahal Systems LiveSQLBench (BIRD-SQL Pro)
Fits text-to-SQL agents on PostgreSQL that must also do writes, and teams that worry about benchmark contamination since databases are added over time. Borrow its soft execution accuracy for SELECT tasks and custom test cases for management tasks.
What it asks the agent to do
- write a PostgreSQL business-intelligence query over a new unseen database
- use hierarchical external knowledge to resolve domain terms
- perform a management task that creates, updates or deletes data
- pass per-question verifiable test cases
What to borrow from it
baselinegrading designcase template
Caveats
No paper, only a repo and dataset card; ground truth and test cases are gated by email; license is stated as MIT in the repo sidebar but CC-BY-SA-4.0 in the README and dataset card; Base-Full v1 is 600 tasks over 22 databases, Base-Lite 270, Large-v1 480.
Details
| Workloads | sql, data analysis |
|---|---|
| Interaction | single turn |
| Environment | database |
| End state checked | Yes |
| Tools | PostgreSQL execution in Docker, external knowledge documents |
| Grading | programmatic check, state diff |
| Size | 600 |
| Data | mixed |
| Personal data | none |
| License | CC-BY-SA-4.0 |
| Harness | own repo |
| Status | active |