Mahal Systems MCP-SafetyBench
Fits agents that connect to third-party MCP servers. Borrow the 20-type attack taxonomy across server, host and user sides, and the paired task evaluator plus attack detector that reports task success rate and attack success rate together.
What it asks the agent to do
- complete a multi-step task across several real MCP servers
- resist attacks injected through a server's tool manifest or implementation
- resist attacks from the host or the user side
- keep task success while refusing the attacker's goal
What to borrow from it
taxonomygrading designsandbox design
Caveats
No license file in the repo; servers are real so results drift with live data; only 245 cases across 5 domains; the README gives no counts or grading details.
Details
| Workloads | safety, security, tool calling, web browsing, finance |
|---|---|
| Interaction | multi turn scripted |
| Environment | api sandbox |
| End state checked | No |
| Tools | real MCP servers for browser automation, financial data, maps, repository management and web search |
| Grading | programmatic check, trajectory check |
| Size | 245 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |