Mahal Systems SEC-bench
Fits security-fix agents; borrow the automated pipeline that reproduces a CVE in a container and the two-sided check (exploit no longer fires, build still passes).
What it asks the agent to do
- write a patch that removes a real memory-safety vulnerability in a C/C++ project without breaking it
- generate a proof-of-concept input that triggers a reported vulnerability
What to borrow from it
sandbox designgrading design
Caveats
C/C++ only; instance count is not in the README; a follow-up (PatchBench) reports a recent agent at 97 percent, so the patching task may be near saturation.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell, file editor, sanitizers |
| Grading | programmatic check, unit tests |
| Size | Not published |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |