Mahal Systems SOP-Bench
Fits agents that must execute a company procedure exactly. Borrow the metric split (execution completion rate times conditional success equals task success) and tool accuracy.
What it asks the agent to do
- follow a written standard operating procedure to reach a decision
- call mock tools in the order the procedure requires
- classify an email, flag content or inspect a record per the procedure
- handle patient intake or dangerous-goods checks step by step
What to borrow from it
case templategrading designtaxonomy
Caveats
Non-commercial license; tools are mocks; procedures and data were partly AI-generated.
Details
| Workloads | workflow automation, customer support, healthcare, finance, manufacturing |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | mock tools per procedure |
| Grading | exact match, programmatic check |
| Size | 1,811 |
| Data | synthetic |
| Personal data | none |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |