Mahal Systems HumanEvalFix (HumanEvalPack)
A tiny bug-fix template: 164 HumanEval problems times six languages with planted bugs; useful only as a quick multi-language smoke test for a repair agent.
What it asks the agent to do
- fix a bug in a short function given failing tests, in six languages
- explain a function and re-synthesize it from the explanation
What to borrow from it
case template
Caveats
Bugs are hand-planted in toy functions, no repository context, and HumanEval derivatives are saturated and memorized.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | unit tests |
| Size | 984 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | frozen |