Mahal Systems FinBen
Fits teams that need a broad baseline for a finance language model behind an agent rather than the agent itself. Borrow its grouping of 36 datasets into 24 task types to check which skills your model is missing.
What it asks the agent to do
- classify sentiment of financial text
- extract entities and numbers from filings
- answer financial question-answering sets
- summarize financial reports
- score credit risk from tabular data
- forecast stock movement from text
What to borrow from it
baselinetaxonomy
Caveats
An aggregate of older public datasets, most single-turn and not agentic; the size of 24 counts task types, not questions; many component sets are saturated or leaked; licenses differ per dataset even though the code is MIT.
Details
| Workloads | finance, knowledge qa, data analysis |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match, programmatic check |
| Size | 24 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |