Mahal Systems OSWorld-Verified
Fits general desktop agents on office and OS tasks; the current reference number for computer-use models. Borrow the parallel AWS runner that finishes a run within an hour.
What it asks the agent to do
- the 369 OSWorld tasks with fixed evaluators, fuzzy matching and backup sites
- office, daily and professional desktop tasks on Ubuntu
What to borrow from it
baselinegrading designsandbox design
Caveats
Top scores (85 to 90 percent in 2026 reports) are past the 72 percent human baseline, so it no longer separates strong agents; use OSWorld 2.0 for headroom.
Details
| Workloads | computer use, documents, spreadsheets, it ops |
|---|---|
| Interaction | single turn |
| Environment | desktop |
| End state checked | Yes |
| Tools | mouse and keyboard on an Ubuntu VM (AWS or Docker), screenshots |
| Grading | state diff, programmatic check |
| Size | 369 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | saturated |