Mahal Systems Agent Security Bench (ASB)
Fits teams comparing defenses across attack stages (query, observation, memory, plan). Borrow the attack-by-stage matrix and the utility-versus-security metric.
What it asks the agent to do
- complete a domain task (finance, legal, counseling, e-commerce) with tools
- resist direct prompt injection in the user query
- resist injection in tool observations
- resist poisoned memory retrieval and backdoored plans
- avoid calling attacker-specific tools
What to borrow from it
taxonomybaseline
Caveats
Tools are simulated and success is whether an attack tool is called, so it measures willingness, not outcomes; the 400 attack tasks reuse 50 agent tasks; tied to the AIOS framework.
Details
| Workloads | safety, security, tool calling, memory |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 400 attack tools and 20 normal tools, simulated inside AIOS |
| Grading | programmatic check, llm judge |
| Size | 400 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |