Mahal Systems TableBench
Fits assistants that answer questions over a single table in a chat. Borrow the 18-category taxonomy under four headings and the per-category metric choice (exact match, exact match with a 10 percent margin, ROUGE-L, pass at 1 for charts).
What it asks the agent to do
- check whether a claim is supported by a table
- compute a number from a table
- find a trend, correlation or statistic in a table
- write code that draws a chart from a table
What to borrow from it
taxonomygrading designbaseline
Caveats
Questions were generated by GPT-4 agents from seed examples and only 30 percent were human-checked; tables are reused from older datasets such as WTQ and TabFact, so leakage is possible; ROUGE-L is a weak check for open-ended answers.
Details
| Workloads | data analysis, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | optional python interpreter |
| Grading | exact match, programmatic check |
| Size | 886 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |