Mahal Systems OSWorld-Human
Fits any desktop agent team that must report latency or cost, not just success. Borrow the weighted efficiency score that combines success with steps relative to a human.
What it asks the agent to do
- score an OSWorld run against the human reference number of steps
- measure end-to-end latency and wasted actions
What to borrow from it
grading design
Caveats
Not a new task set, only human trajectories and a scorer for OSWorld; the repo warns it must not be used for training; license not stated.
Details
| Workloads | computer use |
|---|---|
| Interaction | single turn |
| Environment | desktop |
| End state checked | Yes |
| Tools | OSWorld result directories |
| Grading | trajectory check |
| Size | 369 |
| Data | real public |
| Personal data | none |
| License | unknown |
| Harness | own repo |
| Status | active |