Mahal Systems Tool Decathlon (Toolathlon)
Fits agents connected to many SaaS tools over MCP. Borrow the per-task evaluation scripts that check end state in the real app and the practice of seeding realistic initial states.
What it asks the agent to do
- complete a long task across several real apps via MCP (Notion, Google Calendar, WooCommerce, Kubernetes, BigQuery)
- start from a realistic app state such as a course site with students or a finance sheet
- update records in one app based on data read from another
What to borrow from it
sandbox designgrading designcase template
Caveats
Needs accounts and API keys for many third-party apps (a hosted evaluation service exists); 108 tasks; code license not stated.
Details
| Workloads | tool calling, workflow automation, email calendar, general assistant, devops |
|---|---|
| Interaction | single turn |
| Environment | mixed |
| End state checked | Yes |
| Tools | 604 tools across 32 applications, mostly MCP servers |
| Grading | programmatic check, state diff |
| Size | 108 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |