Mahal Systems WebArena
Fits browser agents that act inside internal web apps (shop admin, issue tracker, forum). Borrow the self-hosted site sandbox and the per-task evaluator mix of exact answer, target URL and page-content program checks.
What it asks the agent to do
- find a product, order or post on a self-hosted site and report the answer
- change data in an online shop admin panel
- open, edit or comment on issues and merge requests in GitLab
- post or reply on a forum
- look up a route or distance on a map
What to borrow from it
sandbox designgrading designcase templatebaseline
Caveats
Reference answers and evaluators had errors (see WebArena-Verified); the Reddit dump contains real usernames; environments are heavy to host and must be reset between runs.
Details
| Workloads | web browsing, retail, devops, workflow automation |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | Yes |
| Tools | browser actions on self hosted sites, scratchpad, map, wiki |
| Grading | exact match, programmatic check, state diff |
| Size | 812 |
| Data | mixed |
| Personal data | possible |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |
Links
Situations it helped establish
- Unanswerable: The data holds no answer, or the tools can't do what was asked.