Mahal Systems ExploitBench
Fits advanced exploit-development or vulnerability-research agents. Borrow the 16 binary capability flags checked by deterministic oracles, which give graded progress instead of a single pass or fail.
What it asks the agent to do
- build an exploit for a real V8 JavaScript engine bug with mitigations on
- climb a ladder from reaching the code, to crash, to memory read and write, to code execution
What to borrow from it
grading design
Caveats
Narrow (V8 only, 41 bugs); episodes run up to 300 turns so it is expensive; the GitHub repo README is still the inspect_evals template, so details live in the paper and inspect page.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | shell access to a patched d8 binary with oracles |
| Grading | programmatic check |
| Size | 41 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |