Mahal Systems FinSearchComp
Fits finance research or market-data agents that must search and then compute. Borrow the three-tier task split (time-sensitive fetch, simple lookup, complex investigation) and the judge prompts with explicit numeric tolerance bands shipped with each question.
What it asks the agent to do
- fetch a current market figure from a live source
- look up a historical financial fact for a named company or index
- run a multi-step investigation that combines several figures with calculation
What to borrow from it
grading designcase templatebaseline
Caveats
The time-sensitive subset has answers that change daily and needs a live data feed to grade; judge prompts are written in Chinese; half the set is Greater China markets; only data and judge prompts are released, no agent runner; the arXiv page lists CC-BY-NC-SA-4.0 while the dataset card says CC-BY-4.0.
Details
| Workloads | finance, research, web browsing, tool calling |
|---|---|
| Interaction | single turn |
| Environment | mixed |
| End state checked | No |
| Tools | web search, financial data APIs or plugins |
| Grading | llm judge |
| Size | 635 |
| Data | real public |
| Personal data | none |
| License | CC-BY-4.0 |
| Harness | none |
| Status | active |