Mahal Systems TaxCalcBench
Fits tax-preparation, benefits-eligibility or any form-filling agent whose output is a set of numbered lines. Borrow its line-by-line strict and lenient (plus or minus 5 dollars) scoring, pass^k over repeat runs, and fully synthetic taxpayer cases.
What it asks the agent to do
- compute a US federal and state income tax return from taxpayer data
- read W-2 and 1099 PDFs and fill Form 1040 lines
- apply filing-status and credit eligibility rules
- use tax tables without arithmetic errors
What to borrow from it
grading designcase templateprivacy pattern
Caveats
Tiny (50 TY25 cases plus 51 legacy TY24 cases); US personal income tax only; cases are simplified versions of the vendor's own engine tests; models solved under a third of returns, so it mostly measures failure.
Details
| Workloads | government, finance, documents |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | optional web search |
| Grading | exact match, pass^k |
| Size | 50 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |