Mahal Systems WASP
Fits web agents that act on sites with user-generated content. Borrow the two-level metric, intermediate attack success (judge sees the agent diverted) versus end-to-end success (final state check), which separates gullibility from damage.
What it asks the agent to do
- complete a user task on a GitLab or Reddit clone
- ignore an injection planted in page text or a URL
- avoid being diverted to the attacker's goal mid-task
- not finish the attacker's goal end to end
What to borrow from it
grading designcase template
Caveats
Only 4 user tasks and 21 attacker goals crossed into 84 cases; non-commercial license; needs the full WebArena docker setup; the utility set's grading method is not described.
Details
| Workloads | safety, security, web browsing |
|---|---|
| Interaction | single turn |
| Environment | browser |
| End state checked | Yes |
| Tools | browser on VisualWebArena GitLab and Reddit clones |
| Grading | state diff, llm judge |
| Size | 84 |
| Data | synthetic |
| Personal data | none |
| License | CC-BY-NC-4.0 |
| Harness | own repo |
| Status | active |