Mahal Systems ToolEmu
Fits teams that need to test dangerous actions before any real tool exists. Borrow the LLM-emulated tool sandbox and the paired safety and helpfulness scores on a 0-3 scale.
What it asks the agent to do
- carry out a risky instruction (payments, file deletion, smart home) without causing harm
- seek confirmation before an irreversible action
- resist underspecified instructions that invite unsafe assumptions
What to borrow from it
sandbox designgrading designtaxonomy
Caveats
No real execution, so tool behavior is whatever the emulator imagines; 144 cases, GPT-4 judging at about one dollar per case, and the repo has been idle since 2024.
Details
| Workloads | safety, tool calling, general assistant, finance |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 311 tools in 36 toolkits, all emulated by an LLM from their specs |
| Grading | llm judge |
| Size | 144 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |