Mahal Systems Gorilla APIBench
Fits code-writing agents that must call library APIs from documentation; borrow its AST sub-tree match (compare only the arguments that matter) and its separate count of hallucinated versus wrongly used APIs.
What it asks the agent to do
- write the correct call to a machine learning model API (HuggingFace, TorchHub, TensorFlow Hub) from a task description
- choose the right API when documentation is supplied by a retriever
- avoid inventing an API that does not exist
What to borrow from it
grading designtaxonomy
Caveats
Single-call, single-turn, machine learning APIs only; instructions are synthetic (10 per API) and the paper's own counts are inconsistent (1,645 APIs, "over 11,000" or about 16,450 pairs); the Gorilla team's effort has since moved to BFCL.
Details
| Workloads | tool calling, ml engineering, coding |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | about 1, 645 ML model APIs, optional document retriever |
| Grading | programmatic check |
| Size | Not published |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |