Mahal Systems MLRC-Bench
Fits research-automation agents; borrow "fraction of the gap between baseline and top human closed" as a metric that avoids an LLM judge.
What it asks the agent to do
- propose and implement a new method for an ML research competition task
- improve on a provided baseline measured by the competition's own metric
What to borrow from it
grading design
Caveats
Only 7 tasks, each costly to run; best agent closed 9.3 percent of the gap, so it mostly measures zero; license for code and data not stated.
Details
| Workloads | ml engineering, research |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, python, gpu |
| Grading | programmatic check |
| Size | 7 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |