HealthBench

Fits patient-facing or clinician-facing chat assistants with no tools. Borrow the per-conversation weighted rubric criteria (48,562 written by 262 physicians), the seven themes and behavior axes, and the Consensus and Hard subsets.

What it asks the agent to do

• answer a health question from a member of the public
• handle a clinician request such as summarizing notes or drafting a message
• recognize an emergency and refer the user
• ask for missing context before answering
• tailor depth and wording to the user's expertise

What to borrow from it

grading designcase templatetaxonomy

Caveats

Conversation-only, no tool use or state; most conversations were synthetically generated from physician-written situations; grading depends on a model grader (GPT-4.1 in the reference code); authored by a model vendor; simple-evals is otherwise deprecated and only keeps HealthBench as a reference implementation.

Details

Links

• Paper
• Github