Mahal Systems MedQA (USMLE)
Context only. Fits nothing agentic; useful as a knowledge floor for a medical assistant and as the source vignettes that AgentClinic and MedCalc-Bench reuse.
What it asks the agent to do
- answer a board-exam style multiple-choice medical question
- pick the most likely diagnosis or next step from a clinical vignette
What to borrow from it
baseline
Caveats
Saturated (frontier models score near the ceiling); exam questions, not clinical work; likely in pretraining data; the 12,723 count is the English subset, Chinese subsets add 34,251 and 14,123.
Details
| Workloads | healthcare, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 12,723 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | saturated |