Mahal Systems SupChain-Bench
Fits order-management, logistics operations and back-office agents that chain many read-only API calls under a procedure. Borrow the oracle-script ground truth (expected tool outputs per question), the entity-level precision and recall scoring, and the three difficulty levels by step count.
What it asks the agent to do
- answer supply chain knowledge questions (multiple choice, true or false)
- diagnose a trade order's status by chaining calls across trade, fulfillment and warehouse systems
- follow a written standard operating procedure for an order issue
- detect cancellation, error or fake-shipping flows
What to borrow from it
grading designsandbox designcase template
Caveats
The 435 count covers the knowledge questions only, the tool-calling task count is not stated; all data is generated by a script with 100 trades by default; tools are read-only so no state change is tested; one SOP only.
Details
| Workloads | manufacturing, workflow automation, tool calling, knowledge qa |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | No |
| Tools | eight read only query tools over synthetic order, fulfillment and warehouse tables |
| Grading | programmatic check, trajectory check, exact match, llm judge |
| Size | 435 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |