Mahal Systems ToolAlpaca
Fits teams that need to test an agent against APIs that do not exist yet or cannot be hit in test; borrow its LLM-simulated API server and its procedure/response/overall grading split. Mostly a training corpus, not a benchmark.
What it asks the agent to do
- read an API description it has never seen and call it correctly
- choose the right sequence of calls without redundant actions
- answer the user's request from the API responses
What to borrow from it
user simulatorsandbox designprivacy pattern
Caveats
Tiny eval set (100 simulated, 114 real instances) graded only by GPT-4; the simulated API responses are made up by a model so they can be wrong; built for 2023-era 7B/13B fine-tunes and effectively saturated for current models.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 10 simulated APIs and 11 real APIs in the eval set; 400+ APIs in training data |
| Grading | llm judge |
| Size | 214 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |