Mahal Systems OSWorld
The reference for desktop agents; borrow the per-task setup config plus execution-based evaluator script pattern and the VM snapshot reset.
What it asks the agent to do
- edit a document, spreadsheet or slide deck in LibreOffice
- change settings or manage files in the OS
- use Chrome, GIMP, VLC, VS Code or Thunderbird to finish a task
- chain actions across several apps
What to borrow from it
sandbox designgrading designcase templatebaseline
Caveats
Original release had over 300 task and environment bugs and is superseded by OSWorld-Verified; VM hosting is heavy; Windows and macOS tasks are a small minority.
Details
| Workloads | computer use, documents, spreadsheets, it ops |
|---|---|
| Interaction | single turn |
| Environment | desktop |
| End state checked | Yes |
| Tools | mouse and keyboard on an Ubuntu VM, screenshots, accessibility tree |
| Grading | state diff, programmatic check |
| Size | 369 |
| Data | real public |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, own repo |
| Status | deprecated |
Links
Situations it helped establish
- Unanswerable: The data holds no answer, or the tools can't do what was asked.