Mahal Systems REALM-Bench
Fits planning and scheduling agents in operations or logistics. Borrow the six metrics (goal satisfaction, makespan, coordination, constraints, resource use, replanning).
What it asks the agent to do
- plan routes or schedules with several agents and dependencies
- replan when a disruption arrives mid-execution
- solve supply-chain and job-shop scheduling instances
- coordinate logistics for an event or disaster response
What to borrow from it
taxonomygrading design
Caveats
Only 11 scenario families (paper says 14); classical OR baselines beat LLMs; no tool environment.
Details
| Workloads | planning, multi agent, manufacturing |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none; planning over given problem instances |
| Grading | programmatic check |
| Size | 11 |
| Data | mixed |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |