Mahal Systems Commit0
Fits greenfield coding agents that build from a spec rather than patch; borrow the "percent of tests passed" partial credit and the lint/type-check gates.
What it asks the agent to do
- implement a whole Python library from its specification and a test suite
- iterate on failing unit tests, lint and type checks until the library works
What to borrow from it
grading designcase template
Caveats
Only 57 libraries, all well-known Python packages whose source is in training data, so memorization is a risk; runs are long and costly.
Details
| Workloads | coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor, pytest |
| Grading | unit tests, programmatic check |
| Size | 57 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |