Mahal Systems AppWorld
Fits assistants that drive several apps through APIs on a user's behalf. Borrow its database-state unit tests that accept any valid path and also check for unwanted side effects.
What it asks the agent to do
- complete a day-to-day errand across several apps (shopping, music, payments, messaging)
- write code that calls app APIs with loops and conditions
- read data from one app and act on it in another
- avoid collateral changes while completing the task
What to borrow from it
grading designsandbox designcase templatebaseline
Caveats
Consumer apps only (no enterprise systems); test-set solutions are shipped encrypted, and the agent must write Python rather than call tools.
Details
| Workloads | general assistant, tool calling, email calendar, shopping, planning |
|---|---|
| Interaction | single turn |
| Environment | api sandbox |
| End state checked | Yes |
| Tools | 457 REST APIs over 9 consumer apps, Python code execution |
| Grading | state diff, unit tests |
| Size | 750 |
| Data | synthetic |
| Personal data | none |
| License | Apache-2.0 |
| Harness | inspect_evals, hal, own repo |
| Status | active |
Links
Situations it helped establish
- Unanswerable: The data holds no answer, or the tools can't do what was asked.