Mahal Systems FinQA
Fits finance agents that compute figures from reports and must show their work. Borrow its two-level grading (execution accuracy on the final number plus program accuracy on the reasoning steps) when you want to check reasoning, not just the answer.
What it asks the agent to do
- answer a numeric question from an earnings report table and text
- produce the step-by-step reasoning program that yields the answer
- pick the right numbers out of a table and the surrounding paragraphs
What to borrow from it
grading designcase templatebaseline
Caveats
From 2021 and widely trained on since, so leakage into current models is likely; questions cover S&P 500 earnings reports only; a private test set exists but needs a CodaLab submission.
Details
| Workloads | finance, documents, data analysis |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, programmatic check |
| Size | 8,281 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |