Mahal Systems MTRAG
Fits retrieval-augmented support or helpdesk assistants that hold a conversation over a knowledge base. Borrow its conversation-plus-corpus case format, its unanswerable and non-standalone question labels, and its separate retrieval and generation scoring.
What it asks the agent to do
- answer the latest turn of a long conversation using a document corpus
- resolve a question that only makes sense given earlier turns
- say the corpus does not contain the answer
- retrieve the right passages from technical documentation, finance or government text
What to borrow from it
case templategrading designuser simulator
Caveats
Only 110 human conversations (842 tasks) and four domains; conversations are scripted, so the agent cannot change the dialogue path; corpora are large downloads that must be indexed before running; a 2026 follow-up (MTRAG-UN, 666 tasks, six domains) extends it.
Details
| Workloads | knowledge qa, customer support, finance, government |
|---|---|
| Interaction | multi turn scripted |
| Environment | static |
| End state checked | No |
| Tools | retriever over four provided corpora |
| Grading | llm judge, human |
| Size | 842 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |