Mahal Systems Long Code Arena
Six project-wide tasks in one suite; the CI build repair task, graded by re-running the real GitHub Actions workflow, is a usable pattern for devops agents.
What it asks the agent to do
- repair a failing GitHub Actions CI build from its logs
- generate code that uses a specific library across a project
- localize a bug, write a commit message, or summarize a module given the whole repo
What to borrow from it
taxonomygrading design
Caveats
Mostly single-turn model tasks rather than agent runs; per-task sizes and licenses were not visible from the paper page, and the leaderboard has not moved much since 2024.
Details
| Workloads | coding, devops |
|---|---|
| Interaction | single turn |
| Environment | mixed |
| End state checked | Yes |
| Tools | none |
| Grading | programmatic check, unit tests, llm judge |
| Size | Not published |
| Data | real public |
| Personal data | possible |
| License | unknown |
| Harness | own repo |
| Status | frozen |