Mahal Systems Multi-SWE-bench
The largest multi-language issue set; fits coding agents that serve polyglot codebases. Borrow the 68-annotator screening rubric and the mini (400) and flash (300) subsets for cost.
What it asks the agent to do
- resolve a GitHub issue in Java, TypeScript, JavaScript, Go, Rust, C or C++
- compile and run a non-Python project's tests inside Docker
What to borrow from it
baselinecase templatetaxonomy
Caveats
Language mix is uneven and compile-heavy languages make runs slow and expensive; results depend heavily on which scaffold (Agentless, SWE-agent, OpenHands) is used.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 1,632 |
| Data | real public |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |