Mahal Systems CRUXEval
Fits teams checking whether a model actually reads code before editing it; borrow the execution-prediction task as a cheap probe inside a bigger code-review suite.
What it asks the agent to do
- predict the output of a short Python function for a given input
- predict an input that produces a given output
What to borrow from it
grading design
Caveats
Functions are 3 to 13 lines of LLM-generated Python; GPT-4 was already at 75 to 81 percent in 2024, so current models are near the ceiling.
Details
| Workloads | coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 800 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | frozen |