Mahal Systems Vending-Bench
Fits long-running operations agents where the question is coherence over weeks of simulated time. Borrow the single hard metric (ending bank balance) and the failure-mode list (forgotten orders, misread schedules, meltdown loops).
What it asks the agent to do
- run a simulated vending business over a long horizon
- order stock, set prices and pay fees without losing track
- stay coherent across millions of tokens of context
What to borrow from it
grading design
Caveats
Closed simulation run by the authors, no public code found; one scenario; runs are very expensive (over 20M tokens) and later versions report models gaming the environment.
Details
| Workloads | planning, finance, retail |
|---|---|
| Interaction | autonomous long horizon |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | simulated email, inventory, pricing and bank tools |
| Grading | programmatic check |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |