Mahal Systems Who&When
Fits teams that want an automated judge to blame the right agent and step after a multi-agent failure. Borrow the agent-plus-step labeling scheme; the reported baselines (about half right on agent, far lower on step) set expectations.
What it asks the agent to do
- read a failed multi-agent run and name the agent responsible
- point to the exact step where the run went wrong
- explain the failure in plain language
What to borrow from it
grading designtaxonomybaseline
Caveats
Small (184 failures) and drawn from CaptainAgent and Magentic-One on GAIA and AssistantBench queries, so it covers a narrow set of orchestration styles.
Details
| Workloads | multi agent, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | 184 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |