Mahal Systems SWE-rebench
Use it when you worry that SWE-bench scores are contaminated: the leaderboard is rebuilt monthly from new issues. Borrow the automated pipeline that turns issues into tested tasks.
What it asks the agent to do
- resolve a freshly mined GitHub issue in a Python repo the model has not seen
- set up an unfamiliar project's environment and run its tests
What to borrow from it
privacy patternbaselinecase template
Caveats
Tasks are filtered by LLM-written quality labels rather than humans, so some are unclear; the dataset is over 21,000 instances but the scored leaderboard subset changes each month.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | Not published |
| Data | real public |
| Personal data | possible |
| License | CC-BY-4.0 |
| Harness | own repo |
| Status | active |