Mahal Systems PersistBench
Fits assistants with persistent user memory. Borrow its two failure classes, cross-domain leakage and memory-induced sycophancy, as test categories.
What it asks the agent to do
- decide whether a stored memory should influence an unrelated new conversation
- avoid agreeing with a user bias that stored memories reinforce
- judge when a long-term memory should be forgotten
What to borrow from it
taxonomycase template
Caveats
No tool use; data and code were promised on publication but no repository found; size unknown.
Details
| Workloads | memory, safety, general assistant |
|---|---|
| Interaction | single turn |
| Environment | static |
| End state checked | No |
| Tools | none; memory and query pairs |
| Grading | llm judge |
| Size | 500 |
| Data | synthetic |
| Personal data | none |
| License | unknown |
| Harness | none |
| Status | active |