Mahal Systems BIRD-Interact
Fits conversational database assistants that must handle vague requests, clarification and write operations. Borrow the function-driven user simulator, the two modes (scripted c-Interact and open a-Interact) and the per-task executable test cases.
What it asks the agent to do
- ask a simulated user to clarify an ambiguous database request
- write and run PostgreSQL queries for business intelligence questions
- create, update or delete rows and tables as the user asks
- recover from an execution error and continue the conversation
- decide when to explore the database versus ask the user
What to borrow from it
user simulatorgrading designcase template
Caveats
Ground-truth SQL and test cases are gated behind an email request; the lite split is 300 tasks in the paper but 270 in the repo README; the dataset is CC-BY-SA-4.0 while code is MIT; best model solves about 17% of full tasks so scores are hard to compare.
Details
| Workloads | sql, data analysis, customer support |
|---|---|
| Interaction | multi turn simulated user |
| Environment | database |
| End state checked | Yes |
| Tools | PostgreSQL execution, knowledge base and metadata files, user simulator |
| Grading | programmatic check, state diff |
| Size | 600 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Ambiguous: The request matches more than one thing, or is too vague to act on without a question or a look at the data.
- Multi-turn pair: The same request arrives over several turns, with details given late or changed, and is scored against its single-turn twin.
- Tool error: A tool call fails, times out or returns an error partway through.