Mahal Systems API-Bank
Fits assistants that must pick one API out of many and fill its arguments from a chat history. Borrow its three-level split (call, retrieve then call, plan then retrieve then call) and its dialogue-plus-expected-call case format.
What it asks the agent to do
- call a known API with the right arguments given the dialogue so far
- search an API store for the right API, then call it
- plan and chain several API calls to serve one request
- write the reply to the user after the call returns
What to borrow from it
case templatetaxonomy
Caveats
Small (314 dialogues, 753 API calls), unchanged since 2023, and evaluation scripts need adapting; the API calls are judged by matching database effects and results, and replies by ROUGE-L, so top models have little headroom left.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | multi turn scripted |
| Environment | api sandbox |
| End state checked | No |
| Tools | 73 hand built runnable APIs backed by a local database, API search tool |
| Grading | programmatic check |
| Size | 314 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |