Mahal Systems τ-bench
Fits support or booking agents that must follow a written policy while talking to a customer and writing to a backend. Borrow the end-state database diff as the pass/fail check, the pass^k reliability metric, and the simulated user prompted with a hidden user instruction.
What it asks the agent to do
- look up an order or flight reservation for a customer
- exchange, return or cancel an order under the written retail policy
- change or cancel a flight booking under the written airline policy
- collect identity details from the user before acting
- refuse a request the policy forbids
What to borrow from it
grading designuser simulatorcase template
Caveats
The repo is marked deprecated and its tasks are not updated (dozens were later found invalid and fixed in the τ²-bench repo); only two domains, 115 retail and 50 airline tasks, all LLM-generated data, and the leaderboard is a static table in the README.
Details
| Workloads | customer support, retail, travel, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | domain APIs over a JSON database (7 write and 8 read tools in retail, 6 write and 7 read in airline) |
| Grading | state diff, pass^k |
| Size | 165 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | hal, own repo |
| Status | deprecated |
Links
Situations it helped establish
- Forbidden or confirm-first: The policy forbids the request, or it needs the person's confirmation before the agent acts.