Mahal Systems RE-Bench
Fits ML research-engineering agents; its value is the 71 human-expert 8-hour attempts that give a time-budgeted human baseline to compare against.
What it asks the agent to do
- optimize a training script or GPU kernel for runtime without changing behavior
- fine-tune or repair a small language model to improve a loss or win rate
- fit a scaling law from limited compute experiments
What to borrow from it
baselinegrading design
Caveats
The repo is archived and marked deprecated; only 7 environments; some files are password-protected and METR asks that solutions not be published.
Details
| Workloads | ml engineering, research |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, python, gpu |
| Grading | programmatic check |
| Size | 7 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | deprecated |