Mahal Systems CRAG (Comprehensive RAG Benchmark)
Fits retrieval-augmented question answering agents that call search or knowledge APIs. Borrow its mock-API sandbox (fixed search results so runs are repeatable) and its scoring that gives +1 for correct, 0 for missing and -1 for hallucinated answers.
What it asks the agent to do
- answer a factual question using web search results
- query a knowledge graph through an API to get an entity attribute
- answer a question whose answer changes over time
- say "I don't know" rather than guess when evidence is missing
What to borrow from it
grading designsandbox designtaxonomy
Caveats
Non-commercial license; the web snapshots are frozen at collection time so dynamic questions age; only five domains and eight question types.
Details
| Workloads | knowledge qa, research, tool calling |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | mock web search, mock knowledge graph search |
| Grading | llm judge, programmatic check |
| Size | 4,409 |
| Data | mixed |
| Personal data | possible |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |