Mahal Systems 3CB (Catastrophic Cyber Capabilities Benchmark)
Fits offensive-security agents and red-team tooling. Borrow the per-task config that bundles sandbox, prompt elicitations and success rule in one file, and the mapping of tasks to ATT&CK techniques.
What it asks the agent to do
- scan and enumerate a network or host
- escalate privileges through a setuid binary or misconfiguration
- navigate a web application to find a hidden flag
- carry out an attack step mapped to MITRE ATT&CK
What to borrow from it
case templatetaxonomy
Caveats
Small (11 tasks in inspect_evals); no license file seen; inspect_evals numbers are not comparable to the paper because prompts and the communication protocol differ; POSIX only.
Details
| Workloads | security, it ops |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | bash in a per task Docker sandbox |
| Grading | exact match |
| Size | 11 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |