Mahal Systems DeepScholar-Bench
Fits literature-review or scientific writing agents. Borrow the live-refresh pipeline that mines new arXiv papers for queries with human-written reference text, and the split metrics for organization, nugget coverage, reference coverage and citation precision.
What it asks the agent to do
- write the related-work section for a recent paper given its topic
- retrieve relevant prior papers from the web
- cite sources so each statement can be verified
What to borrow from it
grading designcase templatebaseline
Caveats
The June 2025 set has only 63 queries (a later 200-query set exists); tasks are limited to related-work sections of arXiv papers; grading uses LLM judges built on the LOTUS framework.
Details
| Workloads | research, science, web browsing |
|---|---|
| Interaction | autonomous long horizon |
| Environment | browser |
| End state checked | No |
| Tools | web search, paper retrieval |
| Grading | llm judge, programmatic check |
| Size | 63 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |