Mahal Systems PaperArena
Fits agents that answer questions over a private document corpus with a tool set (parse, retrieve, compute). Borrow the modular tool platform and the question pipeline with obfuscation and manual review over a fixed corpus of 100 papers.
What it asks the agent to do
- answer a question that needs facts from several papers
- parse PDFs, tables and figures with tools
- run code to compute a value from paper data
What to borrow from it
sandbox designcase templatetaxonomy
Caveats
No license file in the repo; questions were generated by an LLM then checked, so some may be tool-chain shaped rather than user shaped; binary GPT-4o judging; only AI papers.
Details
| Workloads | research, science, tool calling |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | No |
| Tools | PDF parser, figure and table analyzers, cross reference search, web search, paper database, code executor |
| Grading | llm judge |
| Size | 784 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |