Mahal Systems DocMath-Eval
Fits finance or analyst agents that must compute figures from 10-K and 10-Q filings. Borrow its four-way split (simple vs complex reasoning, short vs long document) and its numeric answer checker that accepts chain-of-thought or program-of-thought output.
What it asks the agent to do
- answer a numeric question from a financial filing with text and tables
- find the right table cells across a long document with many tables
- do multi-step arithmetic and return a number
- write a short program that computes the answer
What to borrow from it
grading designcase templatetaxonomy
Caveats
Three of four subsets are re-annotated from older datasets (TAT-QA, FinQA, MultiHiertt, TAT-HQA), so overlap with those is built in; test-split answers were only released in 2026 and the official leaderboard is a CodaLab submission.
Details
| Workloads | finance, documents, data analysis |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | programmatic check |
| Size | 4,000 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |