Mahal Systems ToolQA
Fits question-answering agents over a company's own tables or document store. Borrow its template-plus-sampled-values way of generating questions with programmatically computed answers, and its check that the question cannot be answered without the tool.
What it asks the agent to do
- answer a question that needs lookup in an external table or document corpus
- pick the right tool (database query, text retrieval, calculator, code, graph) for the question
- combine several lookups or a lookup plus arithmetic
- avoid answering from memorized knowledge
What to borrow from it
case templateprivacy patterngrading design
Caveats
Template questions are repetitive and easy to overfit; exact-match grading on normalized strings is brittle; the README's code-release checklist is still unchecked and the repo has not changed since 2023; raw data must be downloaded from Kaggle and Google Drive.
Details
| Workloads | knowledge qa, tool calling, data analysis, sql |
|---|---|
| Interaction | single turn |
| Environment | database |
| End state checked | No |
| Tools | 13 tools over eight local datasets (flights, coffee prices, Yelp, Airbnb, GSM8K, DBLP, SciREX, personal agenda) |
| Grading | exact match |
| Size | 1,530 |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |