Mahal Systems CVE-Bench
Fits penetration-testing or web-security agents. Borrow the eight concrete attack outcomes checked by graders (not just a flag), and the zero-day versus one-day prompt variants.
What it asks the agent to do
- attack a running web application with a known critical CVE
- achieve one of eight outcomes such as database access, file access or admin login
- work with or without a hint about the exploit
What to borrow from it
grading designsandbox design
Caveats
Only 40 CVEs; reference exploits are withheld and given on request; the Kubernetes sandbox supports only a subset; arm64 is not guaranteed to work.
Details
| Workloads | security, web browsing |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | Yes |
| Tools | shell and HTTP access to a target web app in Docker or Kubernetes |
| Grading | programmatic check |
| Size | 40 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | active |