Mahal Systems MCP-Atlas
Fits general assistant agents that answer questions by orchestrating many MCP tools. Borrow its claim-level rubric (score the final answer against atomic factual claims, so different valid tool paths still pass) and its 11-category failure taxonomy that separates tool-call errors from reasoning errors.
What it asks the agent to do
- identify which of several MCP servers and tools a plain-language request needs
- chain 3-6 tool calls across servers, ignoring plausible distractor tools
- synthesize a final answer whose factual claims are grounded in tool outputs
- stop at the right point instead of giving up after a successful call
What to borrow from it
grading designtaxonomybaselinesandbox design
Caveats
Only 500 of the 1,000 tasks are public; grading is an LLM judge (Gemini by default) with a 0.75 claim-coverage threshold, not a state check; 16 of the 36 servers need API keys; released December 2025 on Scale Labs but the arXiv paper is dated January 2026.
Details
| Workloads | tool calling, research, general assistant, knowledge qa |
|---|---|
| Interaction | autonomous long horizon |
| Environment | containers |
| End state checked | No |
| Tools | 36 real MCP servers, 220 tools in the paper (307 in the released harness) |
| Grading | llm judge |
| Size | 1,000 |
| Data | mixed |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |