Mahal Systems SWE-bench Verified
The default number to compare against for issue-to-patch coding agents in Python. Borrow its human screening step, where engineers rejected tasks whose issue text or tests were unfair.
What it asks the agent to do
- read a GitHub issue and edit the repository to resolve it
- make a failing test pass without breaking other tests
- locate the right file and function in a large Python project
What to borrow from it
baselinegrading designcase template
Caveats
Python only, 12 repos, and the tasks predate most current models, so leakage is likely; SWE-EVO reports frontier agents above 70 percent, so headroom is small. HAL runs a Mini subset, not all 500.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 500 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | inspect_evals, hal, own repo |
| Status | active |