Mahal Systems FHIR-AgentBench
Fits read-only EHR question-answering agents that talk to a FHIR server. Borrow the two-stage scoring (resource retrieval recall and precision, then answer correctness) and the single-turn versus multi-turn and text versus code agent variants.
What it asks the agent to do
- answer a clinician's question by querying a FHIR server
- pick the right FHIR resource types and search parameters
- reason over retrieved resources in natural language or by running code
- return no result when no record matches
What to borrow from it
case templategrading designsandbox design
Caveats
Read-only, no orders or writes; requires setting up a Google Cloud FHIR store; built on the 100-patient MIMIC-IV-FHIR demo (PhysioNet, ODbL) so the record space is small; questions are reused from EHRSQL-2024; the best agent reaches only 50 percent.
Details
| Workloads | healthcare, tool calling, knowledge qa |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | No |
| Tools | FHIR REST API on a Google Cloud Healthcare store, Python code interpreter |
| Grading | exact match, programmatic check |
| Size | 2,931 |
| Data | real public |
| Personal data | possible |
| License | CC-BY-4.0 |
| Harness | own repo |
| Status | active |