Mahal Systems CI-Bench
Fits CI-repair or build-fixing agents; borrow the log-to-prompt pipeline and the re-execution in the exact failing container as the pass check.
What it asks the agent to do
- read a failed CI build log and patch the code so the build passes
- re-run the fix in the original BugSwarm container environment
What to borrow from it
sandbox designgrading design
Caveats
Only 100 failures, a demo-track paper, no license in the repo, and best tool scored 36 percent with 2025-era models.
Details
| Workloads | devops, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | programmatic check, unit tests |
| Size | 100 |
| Data | real public |
| Personal data | possible |
| License | unknown |
| Harness | own repo |
| Status | active |