Mahal Systems SWE-bench
Fits any agent that fixes bugs or implements changes in an existing codebase from a written issue. Borrow the FAIL_TO_PASS / PASS_TO_PASS test split and the per-task Docker image pattern.
What it asks the agent to do
- read a GitHub issue and edit the repository to resolve it
- make a failing test pass without breaking other tests
- coordinate changes across several functions or files
What to borrow from it
grading designcase templatesandbox designbaseline
Caveats
The full set has unclear or unsolvable issues (that is why Verified exists), is Python-only, and is in most training sets, so scores are contaminated; most teams now report Verified instead.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 2,294 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | frozen |