Mahal Systems AMA-Bench
Fits long-running agents whose memory must cover their own tool calls and observations, not a human chat. Borrow the idea of mining real trajectories (from ALFWorld, WebArena, SWE-bench, GAIA, Spider 2.0) and writing expert questions over them.
What it asks the agent to do
- answer a question about an earlier step in a long agent run
- report the state of something after it changed during the run
- summarize what the agent did across a long trajectory
- answer questions about machine-generated event streams rather than chat
What to borrow from it
case templategrading design
Caveats
Grading uses a Qwen3-32B judge; the project site rather than a repository hosts the data and its license was not visible; 1,200 of the 3,696 questions are fully synthetic.
Details
| Workloads | memory, general assistant, coding, web browsing |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 3,696 |
| Data | mixed |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |