Mahal Systems MTU-Bench
Fits booking and lookup agents built on slot-filling dialogues. Borrow its 2x2 taxonomy (single or multi turn by single or multi tool) and its metric set (tool selection, parameter accuracy, turn success, task progress) that needs no LLM judge.
What it asks the agent to do
- pick the right tool and fill its parameters from one user turn
- call several tools in the right order for one request
- track slots across a multi-turn booking or lookup dialogue
- handle requests that fall outside the training distribution
What to borrow from it
taxonomygrading designbaseline
Caveats
Test set is only 431 dialogues drawn from old dialogue corpora rewritten by an LLM; evaluation is string match against one gold path, so valid alternatives count as wrong.
Details
| Workloads | travel, general assistant, tool calling |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | 136 tools derived from task oriented dialogue datasets (MultiWOZ, SGD, TaskMaster, MetaLWOZ, ATIS, SNIPS) |
| Grading | exact match, programmatic check |
| Size | 431 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |