Mahal Systems SWE-bench Multilingual
Fits coding agents that must work outside Python; 42 repos across 9 languages including redis, terraform, lucene and tokio. Use it to check that your harness is not Python-specific.
What it asks the agent to do
- resolve a GitHub issue in a C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby or Rust repo
- build and run a project's own test suite in a language other than Python
What to borrow from it
baselinecase template
Caveats
No paper and only one published baseline (Claude 3.7 Sonnet with SWE-agent, 43 percent), so there is little to compare against; small per-language slices.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests |
| Size | 300 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |