Mahal Systems AgentThreatBench
Fits a quick smoke test for email or support agents mapped to the OWASP agentic top 10. Borrow the two independent binary scores per sample and the four outcome profiles (robust, vulnerable, overly cautious, failing).
What it asks the agent to do
- answer questions from a memory store that holds poisoned entries
- triage an inbox without obeying instructions inside emails
- resolve a support ticket without sending sensitive fields to the wrong recipient
- complete the task (utility) and resist the attack (security) at the same time
What to borrow from it
case templategrading design
Caveats
Tiny (24 samples) and covers only 2 of 10 OWASP categories; no paper and no outside validation; attack markers are simple strings.
Details
| Workloads | safety, security, tool calling, email calendar, customer support |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | mock inbox, email categorizer, summary drafter, customer lookup, send message, ticket resolver, memory store |
| Grading | programmatic check |
| Size | 24 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals |
| Status | active |