Mahal Systems WorfBench
Fits agents that produce a plan before executing, such as workflow builders. Borrow the graph matching evaluation (F1 on node chain and on the workflow graph) that rewards correct parallel structure, not just step order.
What it asks the agent to do
- write a step-by-step plan for a how-to task
- lay out a graph of tool calls with parallel branches
- plan steps for a household or web-shop embodied task
- plan an open-ended grounded task before acting
What to borrow from it
grading designtaxonomy
Caveats
Grades the written plan, not execution, and node matching uses a 0.6 embedding similarity threshold that can misjudge paraphrases.
Details
| Workloads | planning, tool calling, embodied, workflow automation |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none at test time; workflows reference tools from ToolBench, ToolAlpaca, Lumos, ALFWorld, WebShop |
| Grading | programmatic check |
| Size | 2,146 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |