Mahal Systems ProjDevBench
Fits agents that build whole programs end to end; borrow the 80/20 split between execution score and code review score and the rule-based checks for forbidden shortcuts.
What it asks the agent to do
- build a complete C++ project from a requirements document
- finish a partially written project so it passes online-judge test cases
- design a system architecture that meets time and memory limits
What to borrow from it
grading design
Caveats
Only 20 problems, mostly C++ and judge-style rather than production code; the LLM review portion is not calibrated against humans.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor |
| Grading | unit tests, llm judge, programmatic check |
| Size | 20 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |