Mahal Systems AgencyBench
Fits long-running coding or deliverable-building agents that take iterative feedback. Borrow the user-simulation agent that supplies feedback and the mixed rule, vision and LLM judges on deliverables.
What it asks the agent to do
- build a working front-end or game from a brief, then revise it on feedback
- implement back-end or code tasks in a sandbox
- research a topic and deliver a written result
- use MCP tools across a long session (about 90 tool calls)
What to borrow from it
user simulatorgrading designbaseline
Caveats
Runs cost about a million tokens per scenario; rubric scoring is largely LLM-judged; results depend strongly on the scaffold used.
Details
| Workloads | general assistant, coding, research, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | containers |
| End state checked | No |
| Tools | Docker sandbox, browser, MCP tools |
| Grading | llm judge, programmatic check |
| Size | 138 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |