Mahal Systems MLE-bench
The standard test for ML-engineering agents; borrow medal-threshold grading against a human leaderboard and the 22-competition Lite split for cheaper runs.
What it asks the agent to do
- solve a Kaggle competition end to end from the data and description
- train a model, tune it, and produce a submission file within a compute budget
What to borrow from it
baselinegrading designsandbox design
Caveats
Full data is 3.3 TB and needs Kaggle credentials; competitions are old and public, so winning solutions are in training data; the official leaderboard is paused.
Details
| Workloads | ml engineering, data analysis |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, python, gpu |
| Grading | programmatic check |
| Size | 75 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |