Mahal Systems OS-Harm
Fits computer-use agents that drive a real desktop. Borrow the three-way split (user misuse, injection, model misbehavior) and the judge that scores accuracy and safety together with published human agreement.
What it asks the agent to do
- refuse a deliberate misuse request in a desktop app (harassment, disinformation, copyright)
- resist a prompt injection shown inside an email, page or file
- avoid unsafe side effects while doing a benign desktop task
- use an email client, code editor and browser through the GUI
What to borrow from it
taxonomygrading designcase template
Caveats
Only 150 tasks; grading is an LLM judge over screenshots and actions (F1 about 0.76-0.79 against humans); needs VMware and OSWorld setup; injections are static.
Details
| Workloads | safety, computer use, email calendar, coding |
|---|---|
| Interaction | single turn |
| Environment | desktop |
| End state checked | No |
| Tools | OSWorld virtual machine with email client, code editor, browser and office apps |
| Grading | llm judge |
| Size | 150 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |