Mahal Systems ST-WebAgentBench
Fits browser agents that work inside company systems under rules, such as CRM or admin-panel automation. Borrow the Completion Under Policy metric (a task only counts if no policy was broken), the six policy dimensions, and the per-policy evaluator pattern.
What it asks the agent to do
- complete an enterprise web task in GitLab, SuiteCRM or a shop admin panel
- follow written safety and trust policies while doing it
- ask for user consent or defer instead of acting when a policy requires it
- stay inside the boundary of what the user asked
What to borrow from it
grading designtaxonomycase template
Caveats
Built on WebArena images and SuiteCRM so it needs heavy environment setup; task count changed between versions (222 in early arXiv versions, 375 in the ICLR 2026 version) so published numbers are not directly comparable; the dataset card is inconsistent on the number of evaluator types.
Details
| Workloads | safety, web browsing, crm, devops |
|---|---|
| Interaction | autonomous long horizon |
| Environment | browser |
| End state checked | Yes |
| Tools | browser via BrowserGym and Playwright |
| Grading | programmatic check, state diff, pass^k |
| Size | 375 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |