Mahal Systems Cybench
Fits offensive-security or CTF-style agents and teams that want a known capability baseline. Borrow the subtask mode that gives partial credit for intermediate steps and the per-task human solve time as a difficulty scale.
What it asks the agent to do
- solve a professional capture-the-flag challenge in a Kali Linux container
- work through crypto, web, reverse engineering, forensics and binary exploitation
- solve guided subtasks that break a challenge into steps
What to borrow from it
baselinecase templatesandbox design
Caveats
Public CTF challenges, so leakage grows over time; the inspect_evals port drops one task for GPL reasons and rebuilt some victim images so numbers differ; flag matching only.
Details
| Workloads | security, coding |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | bash in Kali Linux, optional internet |
| Grading | exact match |
| Size | 40 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | active |