Mahal Systems RAGBench
Fits teams building the judge for a retrieval-augmented agent rather than the agent itself. Borrow its TRACe labels (relevance, utilization, adherence, completeness) at sentence level as a rubric for your own answer grader, and use its 12 sub-datasets to test that grader.
What it asks the agent to do
- judge whether a response is supported by the retrieved passages
- judge whether the retrieved passages are relevant to the question
- judge how much of the relevant context the response used
- judge whether the response covers everything the context offers
What to borrow from it
grading designtaxonomy
Caveats
Labels were produced by GPT-4o, not humans, so it measures agreement with an LLM annotator; responses were generated by older models; no GitHub evaluation code was found, only the Hugging Face dataset; large size comes from pooling existing QA datasets.
Details
| Workloads | knowledge qa, finance, healthcare, legal, customer support |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 100,000 |
| Data | mixed |
| Personal data | none |
| License | CC-BY-4.0 |
| Harness | none |
| Status | active |