Mahal Systems AndroidLab
Fits mobile agents that run without network or logins. Borrow the four metrics: success, sub-goal success, reversed redundancy and reasonable operation ratio.
What it asks the agent to do
- complete an offline task in one of nine apps (finance, calendar, reader, clock, contacts, maps, music, settings, Zoom)
- reach sub-goals in order without redundant steps
What to borrow from it
grading designcase template
Caveats
Small and offline-only; nine apps; bundled with a training set so test leakage is a risk for models trained on it.
Details
| Workloads | mobile |
|---|---|
| Interaction | single turn |
| Environment | mobile |
| End state checked | Yes |
| Tools | Android virtual device, XML or set of marks views |
| Grading | state diff, trajectory check |
| Size | 138 |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |