Mahal Systems ResearchQA
Fits assistants that answer domain research questions at scale. Borrow the method of mining questions and yes/no rubric items from survey articles so a large rubric set exists without expert annotators.
What it asks the agent to do
- give a long-form answer to a scholarly question across 75 fields
- cite relevant papers and explain their findings
- describe limitations and compare approaches
What to borrow from it
grading designcase templatetaxonomy
Caveats
Rubrics are auto-mined, so some items are shallow; rubric coverage rewards breadth over correctness; no repository seen on the arXiv or dataset pages; size is approximate (full split listed as 21.4k).
Details
| Workloads | research, science, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none required, retrieval optional |
| Grading | llm judge, human |
| Size | 21,400 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | none |
| Status | active |