Mahal Systems CORE-Bench
Fits reproducibility or analysis agents; borrow the question-and-answer report.json grading that avoids judging code directly.
What it asks the agent to do
- reproduce a published paper's results from its code capsule and answer questions about them
- set up the environment, run the code, and read figures or tables it produces
What to borrow from it
grading designcase template
Caveats
HAL has declared it solved after a Claude Opus 4.5 run, so it no longer separates current agents; Medium tier needs Docker-in-Docker.
Details
| Workloads | research, coding, data analysis |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | shell, python, R |
| Grading | exact match |
| Size | 270 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, hal, own repo |
| Status | saturated |