Mahal Systems Humanity's Last Exam
Fits general assistants where raw expert knowledge of the underlying model matters. Borrow the short-answer judge prompt and the calibration-error reporting next to accuracy.
What it asks the agent to do
- answer an expert-level closed-book question in math, science or humanities
- pick the right option on a hard multiple-choice question
- state a confidence that matches accuracy
What to borrow from it
baselinegrading design
Caveats
Not an agent benchmark: closed-book expert questions with no tools or multi-step work; some questions have multimodal inputs; scores say little about tool use or task completion.
Details
| Workloads | general assistant, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge, exact match |
| Size | 2,500 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |