Mahal Systems JourneyBench
Fits contact-center agents that must follow a business flow drawn as a graph. Borrow the User Journey Coverage Score (zero unless the tool-call sequence matches exactly, then the fraction of correct parameters) and the graph representation of journeys used to generate scenarios.
What it asks the agent to do
- walk a customer through a multi-step support journey such as a loan application, order issue or telecom request
- call the prescribed tools in the prescribed order with correct parameters
- stay on the business-defined path when the user wanders
What to borrow from it
grading designcase template
Caveats
No code or data released as of the paper; tools are mocked so there is no end state to check; sequence-exact scoring punishes valid alternative orderings; user is GPT-4o.
Details
| Workloads | customer support, telecom, finance, shopping |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | No |
| Tools | 41 mocked tools with pre generated responses |
| Grading | trajectory check |
| Size | 703 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |