Mahal Systems ToolBench (ToolLLM)
Fits agents that must choose from a very large tool catalogue and chain real web APIs; borrow its single-tool / same-category / cross-collection split (G1, G2, G3) for building a tool coverage taxonomy.
What it asks the agent to do
- pick APIs from a RapidAPI catalogue for one instruction
- chain calls across several tools from the same or different categories
- recover when an API fails or returns nothing useful
- produce a final answer from the collected API results
What to borrow from it
taxonomycase template
Caveats
Calls live RapidAPI endpoints through the authors' key server, so results drift as APIs die; the pass and win rates are ChatGPT judgments; instructions were generated by ChatGPT; StableToolBench replaced it for stable scoring.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 16,464 real RapidAPI endpoints across 49 categories via the authors' proxy, API retriever |
| Grading | llm judge |
| Size | 1,100 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | frozen |