Mahal Systems Finance Agent Benchmark (Vals AI)
Fits research agents that answer analyst questions by searching the web and SEC filings. Borrow the nine-category task taxonomy and the per-fact rubric grading (separate correctness and contradiction checks per answer) rather than one overall judge call.
What it asks the agent to do
- look up a figure or statement in a recent SEC filing
- compare a metric across companies or periods
- compute an adjusted or derived financial metric
- judge whether results beat or missed expectations
- build a simple projection or financial model
- summarize market context for a company
What to borrow from it
grading designtaxonomycase templatebaseline
Caveats
Only 50 of 537 questions are public (CC-BY-4.0); 150 more are by request and the 337-question test set is private, so you can run the harness but not reproduce the leaderboard. Rubrics were drafted by GPT-4o. Live web search means answers can drift over time.
Details
| Workloads | finance, research, web browsing, tool calling, documents |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | web search (Tavily), SEC EDGAR search, HTML page parser, recall of earlier results |
| Grading | llm judge, exact match |
| Size | 537 |
| Data | real public |
| Personal data | none |
| License | CC-BY-4.0 |
| Harness | own repo |
| Status | active |