Mahal Systems EHRSQL
Fits text-to-SQL or data-retrieval agents over hospital databases. Borrow the questions collected from 222 real hospital staff, the time-expression templates, the unanswerable question split, and the execution-based F1 that penalizes confident wrong answers.
What it asks the agent to do
- translate a hospital staff question into SQL over MIMIC-III or eICU
- resolve relative time expressions such as last year or since admission
- abstain when a question cannot be answered from the database
What to borrow from it
case templategrading designprivacy pattern
Caveats
Single-shot text-to-SQL, no agent loop; shipped SQLite databases have shuffled patient values and rebuilding from source needs PhysioNet credentialed MIMIC-III and eICU; the MIMIC-IV version (EHRSQL-2024 shared task) lives in a separate repo; annotations were revised in 2026 (v1.5) so older numbers are not comparable.
Details
| Workloads | healthcare, sql |
|---|---|
| Interaction | single turn |
| Environment | database |
| End state checked | No |
| Tools | SQLite database |
| Grading | exact match, programmatic check |
| Size | 24,411 |
| Data | real public |
| Personal data | possible |
| License | CC-BY-4.0 |
| Harness | own repo |
| Status | active |