Mahal Systems KernelBench
Fits performance-engineering agents; borrow fast_p, which counts a task only if it is both correct and at least p times faster, as a two-part pass rule.
What it asks the agent to do
- rewrite a PyTorch operator, fused block or full model as a custom CUDA kernel
- produce a kernel that is correct and faster than the PyTorch baseline
What to borrow from it
grading design
Caveats
Needs specific GPUs so timings are hardware-dependent; levels 1 to 3 have 250 problems and level 4 is unspecified; no official leaderboard.
Details
| Workloads | ml engineering, coding |
|---|---|
| Interaction | single turn |
| Environment | containers |
| End state checked | No |
| Tools | gpu, compiler |
| Grading | programmatic check |
| Size | 250 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |