Mahal Systems LoCoBench-Agent
Fits agents that must navigate large repositories; borrow its efficiency metrics such as files read versus files modified and read-before-write checks.
What it asks the agent to do
- answer questions and make changes in a codebase of 10K to 1M tokens over up to 50 turns
- use file, search and code-analysis tools efficiently across ten languages
What to borrow from it
taxonomygrading design
Caveats
Codebases are synthetically generated, so they lack real-world mess; nine metrics with no single headline number make comparisons across papers hard.
Details
| Workloads | coding |
|---|---|
| Interaction | multi turn scripted |
| Environment | filesystem |
| End state checked | No |
| Tools | file operations, search, code analysis |
| Grading | programmatic check, trajectory check |
| Size | 8,000 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |