Mahal Systems SWE-EVO
Fits agents asked to do long multi-step work rather than one bug; borrow the partial-credit Fix Rate metric and the release-note-to-task construction.
What it asks the agent to do
- move a Python project from one release to the next using the release notes as the spec
- make coordinated changes across about 20 files while keeping hundreds of tests green
What to borrow from it
case templategrading design
Caveats
Only 48 tasks from seven Python projects, so one task swings results; frontier agents score around 25 percent, so it is hard but noisy.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 48 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |