Mahal Systems BIRD-Critic (SWE-SQL)
Fits SQL support or database debugging agents that take a user complaint and a broken query and must return a working one. Borrow its test-case grading on the live database and its variants (Open 570 multi-dialect, PostgreSQL 530, Flash 200, BigQuery 200).
What it asks the agent to do
- fix a user's broken SQL given their issue description and the database
- diagnose why a query returns wrong results or fails on a given dialect
- rewrite SQL across PostgreSQL, MySQL, SQL Server, Oracle or BigQuery
- produce a corrected query that passes hidden test cases
What to borrow from it
grading designcase templatetaxonomy
Caveats
Ground-truth SQL and test cases are withheld and must be requested by email; human score on the single-dialect set is 78.87% versus about 35-39% for the best models; issues are drawn from user posts rather than one company's workload.
Details
| Workloads | sql, coding, it ops |
|---|---|
| Interaction | single turn |
| Environment | database |
| End state checked | Yes |
| Tools | SQL execution in a per dialect database environment |
| Grading | programmatic check, unit tests |
| Size | 570 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |