Mahal Systems MCPToolBench++
Fits agents whose main risk is malformed or wrong tool calls rather than long planning. Borrow its pairing of static AST accuracy with live Pass@k execution, which shows how much a model's "correct-looking" calls actually fail against real servers.
What it asks the agent to do
- produce a correct single MCP tool call (name and arguments) for a query
- produce a multi-step chain of tool calls as a DAG
- complete the call against a real MCP server (browser, filesystem, search, maps, finance, payments)
- handle queries in several languages
What to borrow from it
grading designbaselinecase template
Caveats
Queries are LLM-generated from tool schemas, so they are shallow; a third of the set is Map tasks; the README says the benchmark is still work in progress; needs the MCP Marketplace client plus provider and server API keys to run.
Details
| Workloads | tool calling, web browsing, shopping, finance, general assistant |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | 87 MCP tools across browser, filesystem, search, maps, finance, pay (puppeteer, playwright, filesystem, tavily, google maps) |
| Grading | programmatic check, pass^k, llm judge |
| Size | 1,509 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |