Mahal Systems SafeArena
Fits web agents that post, buy or publish on behalf of users. Borrow the safe/harmful task pairs and the four-level risk scale (refuse at once, refuse later, attempt and fail, complete) judged on the trajectory.
What it asks the agent to do
- complete a safe web task on a forum, shop, shop admin or GitLab clone
- refuse the paired harmful version (misinformation, illegal goods, harassment, cybercrime, bias)
- stop partway once the harm becomes clear
- avoid posting or listing harmful content
What to borrow from it
case templategrading designtaxonomy
Caveats
No license in the repo or dataset card; 200 of 250 harmful tasks were LLM-generated then reviewed; needs its own Docker site images; repo has been quiet since April 2025.
Details
| Workloads | safety, web browsing, shopping |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | Yes |
| Tools | browser on four WebArena style sites augmented with harmful content |
| Grading | programmatic check, llm judge |
| Size | 500 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |