Mahal Systems Mind2Web-SC
Fits teams adding a policy guardrail in front of a web or booking agent. Borrow the rule-plus-profile case format and the split accuracy (right decision, right violation named, right action kept).
What it asks the agent to do
- decide whether a web agent's planned action breaks a user-profile rule (age, license, membership)
- write guardrail code that checks the rule against the user record
- grant or deny the action and name the violated rule
What to borrow from it
grading designcase template
Caveats
Tests a guard model, not the agent; 200 cases with six made-up rules and random user profiles; no license in the dataset repo, which has not changed since 2024.
Details
| Workloads | safety, web browsing |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none for the guard; recorded SeeAct web agent outputs plus a code runner |
| Grading | exact match |
| Size | 200 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | inspect_evals, own repo |
| Status | active |