Mahal Systems MCP-Bench
Fits agents that must discover and chain tools across several MCP servers from an underspecified request. Borrow its split of rule-based checks (tool name valid, schema compliance, call succeeded) from LLM-judged completion and planning scores, and its "description fuzzing" trick for writing tasks that do not name the tool.
What it asks the agent to do
- pick the right tools from a vague instruction without being told tool names
- chain several tool calls across one or more MCP servers
- pass exact parameters from one tool's output into the next
- write a final answer grounded in the intermediate tool outputs
What to borrow from it
grading designcase templatebaseline
Caveats
Only 104 tasks, all synthesized by o4-mini and graded by o4-mini with shuffled rubrics; tasks hit live public servers so results drift; the README badge says Apache 2.0 but there is no LICENSE file at the repo root.
Details
| Workloads | tool calling, research, finance, travel, science |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | No |
| Tools | 28 live MCP servers, 250 tools (finance, travel, scientific computing, paper search, maps, Wikipedia, Reddit, NASA) |
| Grading | programmatic check, llm judge |
| Size | 104 |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |