Mahal Systems τ³-bench
Fits support agents that must answer from a knowledge base as well as act on a backend, and voice agents. Borrow the banking_knowledge domain pattern (retrieval graded against gold read calls) and the voice evaluation setup with simulated callers.
What it asks the agent to do
- answer a banking customer's question by retrieving the right policy document
- handle airline, retail and telecom support tasks by voice in a full-duplex call
- troubleshoot a telecom issue while guiding the user through device steps
- modify an order or booking under policy
What to borrow from it
grading designuser simulatorsandbox design
Caveats
Not a separate paper or repo: it is release v1.0.0 (March 2026) of the tau2-bench repo, adding a 97-task banking_knowledge domain, voice, and 75+ task fixes on top of the τ² domains; the grading was patched in 1.0.1 (scores moved up to 9 points), and the later τ-bench 2.0 release (multilingual, intake domain) again changed default simulator and judge models so numbers across releases are not comparable.
Details
| Workloads | customer support, finance, knowledge qa, telecom, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | mixed |
| End state checked | Yes |
| Tools | domain APIs, document retrieval over 698 banking documents with 12 retrieval strategies, voice providers |
| Grading | state diff, programmatic check, trajectory check, pass^k |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |