How we think about
evaluating memory.
Memory quality is not a single number. These are the dimensions we use to reason about whether a memory system is actually working.
What we measure,
and why.
Retrieval Quality
Does recall surface the knowledge that is actually relevant to the task, by meaning and by exact term?
Governance Correctness
Does the claim lifecycle reflect the real status of knowledge, and do policies apply consistently?
Provenance Integrity
Can a stored claim be traced back to its source, and does the audit record reflect how memory changed?
Conflict Handling
Are conflicting claims surfaced and routed for review rather than silently resolved?
Session Continuity
Does saved context survive a restart and return to a connected agent in a usable form?
Operational Behavior
Do health checks, backups, and retention behave as configured under real operating conditions?
We follow the
field's work.
The research community has proposed benchmarks for long-conversation and long-term memory, including LoCoMo and LongMemEval. These are external efforts, and we track them because they frame useful questions about what memory systems should be able to do.
Benchmark scores are only as meaningful as the methodology behind them. A number without a described setup, dataset, and scoring method is not evidence. We treat published results as a starting point for evaluation rather than a substitute for it.
No numbers
without method.
This page does not publish internal benchmark numbers. We would rather show nothing than show a figure that cannot be reproduced or that was measured under conditions we cannot describe.
When we have results we can stand behind, they will appear here with the methodology, dataset, and scoring approach attached. Until then, the honest position is that evaluation is ongoing.
Evaluating memory
for your use case?
Tell us what you need to measure and the conditions that matter to your team. We will discuss the evaluation approach that fits.