Mahal Systems StableToolBench
Fits agents that call many third-party web APIs and need a replayable test bed; the cache plus LLM-simulated-API server is the pattern to copy when the real APIs are flaky, and the solvable-only filtering shows how to prune unanswerable cases.
What it asks the agent to do
- pick and chain RapidAPI-style tools for one instruction
- handle multi-tool tasks across categories
- produce a final answer judged solved or unsolved
What to borrow from it
sandbox designgrading design
Caveats
Grading is still a GPT-4 judge (solvable pass rate and win rate) with no state check, API responses on cache misses are model-simulated rather than real, and the 765 tasks are a filtered subset of ToolBench's ChatGPT-written instructions.
Details
| Workloads | tool calling, general assistant |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | virtual API server (response cache, then real API, then GPT 4 simulated response), MirrorAPI model |
| Grading | llm judge |
| Size | 765 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Tool error: A tool call fails, times out or returns an error partway through.