Mahal Systems FRAMES
Fits research or multi-hop retrieval agents that must chain facts across documents. Borrow its per-question reasoning-type labels and the inspect_evals Socrates scoring, which discounts accuracy by extra retrieval hops and penalizes requests for documents that do not exist.
What it asks the agent to do
- answer a question that needs facts from 2 to 11 Wikipedia articles
- combine numeric, tabular or date reasoning across sources
- request only the documents needed and stop retrieving when done
What to borrow from it
grading designcase template
Caveats
Only 824 questions, all over Wikipedia, so domain transfer is limited; the inspect_evals implementation fetches live Wikipedia pages at run time, so content can drift from the answers; grading details in the paper were not confirmed from the abstract.
Details
| Workloads | knowledge qa, research |
|---|---|
| Interaction | single turn |
| Environment | mixed |
| End state checked | No |
| Tools | document request tool (Socrates variant only) |
| Grading | programmatic check, trajectory check |
| Size | 824 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals |
| Status | active |