Mahal Systems ECom-Bench
Fits e-commerce after-sales support agents. Borrow the persona files (consumption habits, personality) that drive the simulated shopper, and the three-part rule-based reward (end database state, required lookup tools called, required keywords in the reply) on top of the τ-bench harness.
What it asks the agent to do
- resolve an after-sales issue for a home or appliance product with a persona-driven shopper
- look up the order and product information before answering
- update the order or ticket state as the task requires
- interpret an image the customer sends (18 of the tasks are multimodal)
What to borrow from it
user simulatorgrading designprivacy pattern
Caveats
Only 53 tasks, so differences between models are noisy; built on τ-bench code; data was LLM-synthesized from historical commercial dialogues and the paper does not describe anonymization, so treat personal data as possible.
Details
| Workloads | customer support, shopping, retail, tool calling |
|---|---|
| Interaction | multi turn simulated user |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | e commerce order and product lookup and update APIs over a local database |
| Grading | state diff, programmatic check, pass^k |
| Size | 53 |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |