Mahal Systems PaperBench
Fits research-replication agents; borrow the rubric tree (8,316 leaf requirements) and the Code-Dev variant that grades code only and skips the GPU reproduction step.
What it asks the agent to do
- reproduce an ICML 2024 paper from scratch: write the code, run the experiments, match the results
- build a codebase against a hierarchical rubric co-written with the paper authors
What to borrow from it
grading designcase template
Caveats
Only 20 papers, each run takes many hours and GPUs; grading relies on an LLM judge (SimpleJudge) whose accuracy is itself benchmarked but not perfect.
Details
| Workloads | ml engineering, research |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, python, gpu |
| Grading | llm judge |
| Size | 20 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |