Mahal Systems AgentAuditor / ASSEBench
Fits teams building an LLM judge for agent safety rather than testing the agent itself. Borrow the human-labeled records with ambiguity flags and the strict/lenient split to calibrate a judge's F1 before trusting it.
What it asks the agent to do
- read a recorded agent interaction and label it safe or unsafe
- spot a security threat such as injection or data leak in a trajectory
- apply a strict or lenient standard to an ambiguous case
- explain which risk type and behavior mode applies
What to borrow from it
grading designtaxonomy
Caveats
It benchmarks judges, not agents; the GitHub repo is a placeholder and the data lives on Hugging Face; records were generated by three LLMs on top of other benchmarks, so it inherits their synthetic scope.
Details
| Workloads | safety, security, tool calling |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none; the evaluator reads recorded trajectories |
| Grading | exact match |
| Size | 2,293 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |