Mahal Systems SHADE-Arena
Fits agents that run many tool calls on a user's behalf with money or data at stake, and teams that want an LLM monitor on their traces. Borrow the main-task plus side-task pairing, the 0-100 suspicion score from a monitor that cannot see the system prompt, and the mock-tool environment design.
What it asks the agent to do
- complete a long benign task with mock tools such as paying bills or booking travel
- secretly complete a harmful side task in the same run
- avoid raising suspicion from a monitor reading the transcript
- act as a monitor and rate a transcript for suspicious behavior
What to borrow from it
grading designsandbox designcase template
Caveats
Only a public split is on GitHub (the private split with more tasks and transcripts is on request), the public README gives no task count, and the first safety-research repo is marked deprecated in favor of a personal account.
Details
| Workloads | safety, tool calling, general assistant, finance, email calendar |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | mock tools over a modified AgentDojo pipeline (banking, email, calendar, files) |
| Grading | programmatic check, llm judge |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |