Mahal Systems MemBench
Fits personal assistants that build a profile of a user over time. Borrow the split into factual versus reflective memory and participation versus observation, and the memory capacity and read/write time metrics.
What it asks the agent to do
- recall a fact from a user's past dialogues
- draw a reflective conclusion about the user from many interactions
- remember events the agent only observed rather than took part in
What to borrow from it
grading designtaxonomy
Caveats
All questions are multiple choice, which is easier than free recall; the paper's tables give only rounded counts (tens of thousands of questions) so the exact size is unclear.
Details
| Workloads | memory, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | exact match |
| Size | Not published |
| Data | synthetic |
| Personal data | none |
| License | MIT |
| Harness | own repo |
| Status | active |