Mahal Systems MedHELM
Fits any medical assistant whose team needs a map of clinical task categories. Borrow the clinician-validated taxonomy (5 categories, 22 subcategories, 121 tasks), the LLM-jury scoring validated against clinician ratings, and the public, gated and private access tiers.
What it asks the agent to do
- support a diagnosis or treatment plan from a patient record
- generate a clinical note, report or care plan
- write patient education or a patient-provider message
- answer a medical research or literature question
- handle an administrative task such as scheduling or billing
What to borrow from it
taxonomygrading designbaseline
Caveats
A suite of 35 existing and new benchmarks, none agentic; many scenarios are gated (PhysioNet, Redivis) or private to Stanford Health Care so outsiders can only run the public subset; leaderboard is not open to submissions; Python 3.10 only.
Details
| Workloads | healthcare, documents, knowledge qa, research |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge, exact match, programmatic check |
| Size | Not published |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |