Mahal Systems AndroidWorld
Fits mobile app agents; borrow parameterized tasks whose setup, success check and teardown read the device state, giving unlimited variants of 116 templates.
What it asks the agent to do
- create or edit a note, contact, event or expense in an Android app
- change a phone setting
- send a message or set an alarm with randomized details
What to borrow from it
grading designcase templatesandbox design
Caveats
Leaderboard agents exceed 90 percent, so it no longer separates strong systems; open source apps only, no e-commerce or enterprise chat; MobileWorld is the successor.
Details
| Workloads | mobile, general assistant |
|---|---|
| Interaction | single turn |
| Environment | mobile |
| End state checked | Yes |
| Tools | Android emulator (touch, type, ADB), 20 apps |
| Grading | state diff, programmatic check |
| Size | 116 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | own repo |
| Status | saturated |