DABstep

Fits finance or operations analytics agents that answer questions from files plus policy documents. Borrow its factoid grading without a judge model (numeric tolerance, fuzzy strings, element-wise lists) and its approach of making synthetic data with real-shaped business context.

What it asks the agent to do

• answer a factual question by combining CSV and JSON files with markdown documentation
• read payment-processing rules in documentation and apply them to the data
• write and run Python code over several files in multiple steps
• return one exact value, string or list as the final answer

What to borrow from it

grading designcase templateprivacy pattern

Caveats

Ground truth is hidden and scored through a submission form; the payments data and business context are synthetic with inflated fraud rates; no dedicated GitHub repo, baseline code lives in a Hugging Face space; best hard-set score around 14-16%.

Details

Links

• Paper
• Leaderboard