Mahal Systems ClassEval
Fits code generators that emit whole modules; borrow class-level versus method-level pass@1 reporting to show where multi-part outputs break.
What it asks the agent to do
- generate a whole Python class with several interdependent methods from a skeleton
- keep methods consistent with each other when generated one at a time
What to borrow from it
grading design
Caveats
Only 100 hand-written classes (410 methods); data is CC BY-NC 4.0 (non-commercial); 2023 vintage and likely memorized.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 100 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | frozen |