Mahal Systems CyberGym
Fits code-security or bug-finding agents that work on real repositories. Borrow the execution oracle that runs the agent's input against a vulnerable and a fixed build, and the four difficulty levels that vary how much context is given.
What it asks the agent to do
- read a vulnerability description and a large C or C++ codebase
- write a proof-of-concept input that triggers the crash
- check the input against the vulnerable and the patched build
What to borrow from it
grading designsandbox designbaseline
Caveats
Data is about 240 GB and the full server data about 10 TB; sources are public OSS-Fuzz and ARVO crashes so models may have seen them; the inspect_evals port had a scoring bug fixed in version 2-A so older results are invalid.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | bash in a Docker sandbox, proof of concept submission script |
| Grading | programmatic check |
| Size | 1,507 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | active |