Mahal Systems MedAgentBench
Fits agents that read and write an electronic health record through standard APIs (chart review, order entry, documentation helpers). Borrow the Docker FHIR sandbox, the 10 task categories written by physicians, and the rule-based checks on POST payloads.
What it asks the agent to do
- look up a patient's lab results, vitals or medications through FHIR queries
- compute a derived value from a chart such as an average or a trend
- place an order or record an observation by POSTing to the FHIR server
- carry out a short clinician-written instruction over one patient's record
What to borrow from it
sandbox designcase templategrading design
Caveats
Patient profiles are real Stanford (STARR) records that were de-identified and time-jittered, not synthetic; grading checks the request payload rather than diffing server state; the reference solutions file must be fetched separately from a Stanford Box link; only 300 tasks and the best model already scores about 70 percent.
Details
| Workloads | healthcare, tool calling, workflow automation |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | No |
| Tools | FHIR REST API (GET and POST) on a HAPI FHIR server in Docker |
| Grading | programmatic check |
| Size | 300 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |