Mahal Systems SWE-bench Lite
A cheap smoke test for a coding agent: single-file edits, at most three hunks, no images or links in the issue. Borrow its filter rules when you need a small, clean subset of a bigger set.
What it asks the agent to do
- fix a single-file bug described in a short GitHub issue
- make a failing test pass without breaking other tests
What to borrow from it
case templatebaseline
Caveats
Filtered to easy single-file cases, so it over-estimates real ability; superseded by Verified for reporting and the leaderboard moves slowly.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 300 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | frozen |