Mahal Systems SWE-PolyBench
Fits coding agents whose work mixes bug fixes, features and refactors; borrow the task-category labels and the file and node retrieval metrics as a partial-credit signal.
What it asks the agent to do
- fix a bug, add a feature, or refactor code in a Java, JavaScript, TypeScript or Python repo
- find the right files and syntax-tree nodes to change
What to borrow from it
taxonomygrading designbaseline
Caveats
Heavily skewed to JavaScript/TypeScript (about 1,700 of 2,110); use the PB500 or the 382-task Verified subset for balanced numbers.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests, programmatic check |
| Size | 2,110 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |