Mahal Systems OSWorld 2.0
Fits long-running desktop agents; the successor to OSWorld-Verified with headroom. Borrow binary completion plus partial-credit scoring and the safety-sensitive action audit.
What it asks the agent to do
- finish an hours-long professional workflow across several desktop apps
- reason over input files and a stateful user profile
- keep going through streaming or changing environments
What to borrow from it
baselinecase templategrading design
Caveats
Very expensive (about 318 tool calls per task); task data gated on Hugging Face; best score 20.6 percent binary; v2.1 is recommended over 2.0.
Details
| Workloads | computer use, documents, data analysis, workflow automation |
|---|---|
| Interaction | autonomous long horizon |
| Environment | desktop |
| End state checked | Yes |
| Tools | Ubuntu VM, desktop apps, files, up to 500 steps |
| Grading | state diff, programmatic check |
| Size | 108 |
| Data | mixed |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | active |