Mahal Systems SWE-bench Multimodal
Fits agents that work on front-end or visualization code where the bug report is a picture. Borrow the way image inputs are attached to an otherwise text-only task.
What it asks the agent to do
- fix a visual bug in a JavaScript library from an issue that includes a screenshot
- read an image in an issue or test and change front-end code accordingly
What to borrow from it
case templategrading design
Caveats
JavaScript only, 17 libraries; the README says 480 tasks are runnable locally and the rest are a hidden split, so published numbers depend on which split was used.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor, image viewing |
| Grading | unit tests |
| Size | 617 |
| Data | real public |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |