Mahal Systems SWE-bench Pro
Fits teams whose coding agent works on larger changes than SWE-bench; the held-out commercial split is the pattern to copy if you want a private test set that nobody can train on.
What it asks the agent to do
- implement a multi-file change (10 to hundreds of lines) from an issue in a large repo
- resolve an issue in Go, Python, JavaScript or TypeScript without breaking existing tests
- work in a codebase the model is unlikely to have seen (copyleft or private)
What to borrow from it
baselineprivacy patterncase template
Caveats
Two versions (v1 with 731 tasks, v2 with 642 re-verified tasks) are not comparable; the commercial and held-out splits are private, and Inspect support is via the Harbor package.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 731 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |