Mahal Systems Windows Agent Arena
Fits agents that must run on Windows desktops. Borrow the Azure-parallel runner that finishes the benchmark in about 20 minutes.
What it asks the agent to do
- change a Windows setting or manage files in Explorer
- edit a document in LibreOffice, Notepad or VS Code
- use Edge or Chrome, VLC, Paint, Clock or Calculator
What to borrow from it
sandbox designbaseline
Caveats
Small; adapted from OSWorld so task style is similar; needs a Windows evaluation image; launch score 19.5 percent versus 74.5 for humans.
Details
| Workloads | computer use, documents, it ops |
|---|---|
| Interaction | single turn |
| Environment | desktop |
| End state checked | Yes |
| Tools | mouse and keyboard on a Windows 11 VM, screenshots, UIA accessibility |
| Grading | state diff, programmatic check |
| Size | 154 |
| Data | real public |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |