Mahal Systems BigCodeBench
A function-level baseline for library-heavy Python; borrow its high-coverage test writing (about 5.6 tests per task, 99 percent branch coverage).
What it asks the agent to do
- write a Python function that combines calls to several libraries from a docstring
- follow a short natural-language instruction to produce a tested function
What to borrow from it
baselinegrading design
Caveats
The repository was archived in July 2026, so it will not be updated; single-function, no repo context; the 148-task Hard subset is the only part with headroom.
Details
| Workloads | coding, tool calling |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 1,140 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | frozen |