Mahal Systems DevBench (DevEval)
Fits agents covering more than coding, such as design docs and test writing; borrow its lifecycle stages as a taxonomy for an agent's golden set.
What it asks the agent to do
- write a software design document from a product requirements document
- set up the environment and implement a small repo in Python, C/C++, Java or JavaScript
- write acceptance and unit tests for the implementation
What to borrow from it
taxonomycase template
Caveats
Only 22 repos, so results are noisy; the design-document stage is LLM-judged; data is CC BY 4.0 while code is Apache-2.0.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests, llm judge |
| Size | 22 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |