Mahal Systems MCPMark
Fits agents that write to Notion, GitHub or databases through MCP. Borrow the per-task verify.py against a curated initial state and the pass^k reliability reporting.
What it asks the agent to do
- create, update and reorganize pages and databases in Notion
- manage issues, branches and files in a GitHub repository
- run queries and change data in a PostgreSQL database
- edit files on a filesystem
- automate a web page with Playwright
What to borrow from it
grading designsandbox designcase template
Caveats
Needs live Notion and GitHub accounts; a Verified subset reports different numbers from the paper; mostly developer-flavored apps.
Details
| Workloads | tool calling, workflow automation, documents, sql, coding |
|---|---|
| Interaction | single turn |
| Environment | mixed |
| End state checked | Yes |
| Tools | MCP servers for Notion, GitHub, Filesystem, PostgreSQL, Playwright |
| Grading | programmatic check, state diff, pass^k |
| Size | 127 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |