Mahal Systems LiveMCPBench
Fits agents that must find the right tool in a large catalog before using it; the paper attributes about half of failures to retrieval. Borrow its Dockerized key-free tool suite and its key-point rubric judged by an LLM against live outcomes.
What it asks the agent to do
- retrieve the right few tools from a catalog of 527 across 70 servers
- compose several tools from different servers for a daily-life task (office, travel, shopping, finance)
- use live online data (news, weather, prices) in the answer
- produce a result that satisfies human-written key points
What to borrow from it
sandbox designgrading designtaxonomy
Caveats
Only 95 tasks; answers depend on live web data so ground truth changes and the grader is an LLM judge (DeepSeek-V3, ~80% human agreement); the leaderboard is a Google Sheet.
Details
| Workloads | tool calling, general assistant, research, travel, shopping |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | 70 MCP servers, 527 tools, no API keys required (LiveMCPTool) |
| Grading | llm judge |
| Size | 95 |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |