Mahal Systems REAL
Fits consumer browser agents that touch money or bookings. Borrow the replica-site sandbox with state checks for actions and rubric judging for lookups.
What it asks the agent to do
- buy, book or order on replicas of Amazon, Airbnb, United, DoorDash, Uber, OpenTable
- send mail or set events on Gmail and Calendar replicas
- find information on LinkedIn, Upwork or Zillow replicas
What to borrow from it
sandbox designgrading design
Caveats
Non-commercial license; replicas are vendor-built clones so fidelity is unverified; small; top score 41 percent at release.
Details
| Workloads | web browsing, shopping, travel, email calendar |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | Yes |
| Tools | deterministic website replicas in a browser |
| Grading | state diff, llm judge |
| Size | 112 |
| Data | synthetic |
| Personal data | none |
| License | CC-BY-NC-SA-4.0 |
| Harness | own repo |
| Status | active |