Persistence
Does a stable preference survive across unrelated turns and delayed use?
PersistBench
A contamination-resistant benchmark for the decisions an agent makes across extended interaction. It measures whether memory changes behaviour when preferences evolve, facts are corrected and old context becomes unsafe to apply.
Motivation
Static probes reward retrieval from a fixed history. Long-running agents must decide what to retain, update and consolidate, and when an old memory should stop influencing the next action. A stored fact earns little credit until it improves a later decision.
Capability ladder
- Naive RAG
- Mem0
- A-MEM
- MEM1
The initial comparison uses LongMemEval and LoCoMo alongside PersistBench tasks. A reinforcement-learning controller is being developed to coordinate short- and long-term memory under one learned policy.
Evaluation design
Correction
Does a newer fact replace an older belief in later behaviour?
Selective forgetting
Can the agent stop applying stale or explicitly withdrawn information?
Conflict handling
Does it identify contradictory memories before taking an action?
Decision effect
Does memory improve the final choice, tool call or escalation?
Evidence trace
Can an evaluator reconstruct which memory influenced the action?
Contamination resistance
Task instances separate behavioural structure from surface content. Details can be regenerated while preserving the capability under test, reducing the value of memorising a public answer key.
Status
Specification and task templates are in preparation. The repository link will be added when the evaluation package is ready for public use.
Academic collaboration: taha@tahayasinbhatti.com