Mahal Systems APPS
Still used as a large, cheap code-generation baseline; its test-case checker is reused by LiveCodeBench, so it is a known-good grading script.
What it asks the agent to do
- turn a natural-language problem statement into a Python program
- pass hidden test cases at introductory, interview and competition levels
What to borrow from it
baseline
Caveats
From 2021 and fully in training data; some test cases are known to be wrong; not an agent benchmark.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 10,000 |
| Data | real scraped |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | frozen |