Mahal Systems LongMemEval
Fits chat assistants with long-term memory across sessions; the five ability categories (extraction, multi-session reasoning, temporal, updates, abstention) are a ready taxonomy for a memory test set. Borrow the needle-in-history construction and the 30 abstention cases.
What it asks the agent to do
- extract a fact the user mentioned in an earlier session
- reason across several sessions to answer a question
- answer a question that depends on when something was said
- give the updated answer after the user changed a fact
- abstain when the history does not contain the answer
What to borrow from it
case templategrading designtaxonomy
Caveats
Filler sessions come from ShareGPT and UltraChat, so real user text may be present; grading depends on a GPT-4o judge; the S variant is about 115k tokens so it is costly to run.
Details
| Workloads | memory, knowledge qa, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none |
| Grading | llm judge |
| Size | 500 |
| Data | mixed |
| Personal data | possible |
| License | MIT |
| Harness | own repo |
| Status | active |