Mahal Systems MedCalc-Bench
Fits clinical decision-support assistants that must compute scores or doses from notes. Borrow the tolerance-band grading (plus or minus 5 percent for decimal outputs), the relevant-entities field for partial-credit diagnosis, and the 55 calculator taxonomy.
What it asks the agent to do
- extract the needed values from a patient note
- choose the right clinical calculator or scoring rule
- compute the value (dose, score, risk, date) and report it
What to borrow from it
grading designcase template
Caveats
Single-turn question answering, not agentic; notes come from published PMC case reports, clinician-written vignettes and templates, so no live record; the authors now point to a corrected successor (MedCalc-Bench Verified) so v1 labels contain known errors; the 1,100 figure is the test split, a 10,543-instance training split also ships.
Details
| Workloads | healthcare, knowledge qa, data analysis |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none in the standard setting; a code interpreter variant exists |
| Grading | exact match, programmatic check |
| Size | 1,100 |
| Data | mixed |
| Personal data | none |
| License | CC-BY-SA-4.0 |
| Harness | inspect_evals, own repo |
| Status | active |