Mahal Systems EHRAgent (EHRSQL-derived agent evaluation)
Fits data-retrieval agents that let clinicians ask questions of structured EHR tables in plain language. Borrow the code-as-plan action format, the interactive debugging loop, and the question complexity levels (number of tables joined).
What it asks the agent to do
- answer a clinician's question over multi-table EHR data by writing and running code
- find and join records across patient, admission, lab and prescription tables
- compute aggregates such as counts, averages or survival rate
- repair generated code from execution error messages
What to borrow from it
case templatebaseline
Caveats
A method paper whose evaluation set is a re-processed slice of EHRSQL (580 MIMIC-III, 580 eICU, 996 TREQS questions) distributed through a Google Drive link; the underlying MIMIC-III and eICU databases need PhysioNet credentialed access; no license file in the repo; success rate is a single exact-match answer check.
Details
| Workloads | healthcare, sql, data analysis, tool calling |
|---|---|
| Interaction | autonomous long horizon |
| Environment | database |
| End state checked | No |
| Tools | Python code executor, table lookup and SQL helper tools, medical knowledge lookup |
| Grading | exact match |
| Size | 2,156 |
| Data | real public |
| Personal data | possible |
| License | unknown |
| Harness | own repo |
| Status | active |