Mahal Systems τ²-bench
Fits technical-support agents where the customer must act on their own device or account (dual control), and any policy-bound support agent that writes to a backend. Borrow the compositional task generator (atomic subtasks with init, solution and assertion functions), the assertion-based end-state check, and the error split between reasoning and communication.
What it asks the agent to do
- troubleshoot a customer's phone service, mobile data or MMS problem by telling the user which steps to take on their device
- look up and modify an order or flight reservation under policy
- guide a user who holds their own tools through a multi-step fix
- refuse or escalate when the policy forbids the request
What to borrow from it
grading designuser simulatorcase templatesandbox design
Caveats
Size 279 is the sum of the paper's per-domain counts (115 retail, 50 airline, 114 telecom sampled from 2,285 generated); the repo has since moved on to τ³-bench and τ-bench 2.0 releases whose scores are not comparable with 1.x, and a 2026 TASTE paper reports frontier models near saturation on the verified task set.
Details
| Workloads | customer support, telecom, retail, travel, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | agent side domain APIs plus user side device tools in telecom (dual control) |
| Grading | state diff, programmatic check, pass^k |
| Size | 279 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | inspect_evals, own repo |
| Status | active |
Links
Situations it helped establish
- Forbidden or confirm-first: The policy forbids the request, or it needs the person's confirmation before the agent acts.
- Multi-turn pair: The same request arrives over several turns, with details given late or changed, and is scored against its single-turn twin.
- Person does a step: Part of the job can only be done by the person, on their own device or account, with the agent guiding.