Mahal Systems CAIBench
Fits teams building both attack and defense agents who want one place to measure knowledge, CTF and range tasks. Borrow the split into five categories and the attack-defense setup where two agents face each other.
What it asks the agent to do
- solve jeopardy-style capture-the-flag challenges
- attack and defend services at the same time in an attack-defense CTF
- run a multi-step exercise in a cyber range
- answer security knowledge questions
- handle personal data without leaking it
What to borrow from it
taxonomysandbox design
Caveats
The CAI repo is archived and read-only; license mixes MIT with proprietary research-only additions; the abstract says over 10,000 instances but no exact count was seen; privacy category uses personal-data-like content.
Details
| Workloads | security, knowledge qa, multi agent |
|---|---|
| Interaction | autonomous long horizon |
| Environment | mixed |
| End state checked | Yes |
| Tools | shell and security tools through the CAI framework |
| Grading | exact match, programmatic check |
| Size | Not published |
| Data | mixed |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | frozen |