Mahal Systems ToolSandbox
Fits assistant agents whose tools have hidden state dependencies (one call only works after another). Borrow the milestone and minefield grading: required world-state checkpoints in a partial order, forbidden events that zero the score, and similarity scoring of each checkpoint.
What it asks the agent to do
- change a phone setting such as wifi or cellular before completing the user's request
- send a message, add a contact or set a reminder that depends on earlier tool results
- ask the user for missing information instead of guessing
- canonicalize fuzzy user input (dates, names) into the form a tool needs
What to borrow from it
grading designsandbox designuser simulatortaxonomy
Caveats
Scenarios are hand-written by two annotators rather than drawn from real traffic; the license is a custom Apple permissive license with no SPDX id; search tools need a paid RapidAPI key; no leaderboard, and no activity signal after the April 2025 paper revision.
Details
| Workloads | tool calling, general assistant, mobile |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | 34 stateful phone assistant tools across 11 domains (settings, contacts, messaging, reminders, search) |
| Grading | state diff, trajectory check |
| Size | 1,032 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |