Two Rules for Evaluating LLM Memory Retrieval

If you build long-term memory for an assistant, you eventually have to answer a question that sounds simple: is the retrieval any good?

Recall and precision are the obvious instruments, and they're necessary. They are also not sufficient, in a way that took me a while to see clearly. Two design decisions did more for the usefulness of my evaluation than any metric choice, and both of them are about what the numbers are allowed to hide.

Rule one: the labeller never sees what retrieval returned

To measure recall you need a relevant set — the facts that should have been retrieved for a given query. Someone or something has to decide what belongs in it.

The tempting way to build that is to run retrieval, look at what came back, and mark each result right or wrong. It's fast, the output is right there, and it feels like labelling.

It isn't. It's grading with the answer key open, and it corrupts the evaluation in a particular direction.

When you see retrieval's output first, it anchors you. A borderline result that came back looks defensible, so you mark it relevant. A fact that should have come back but didn't isn't in front of you, so it never enters the relevant set at all. Both errors push the score in the same direction — up — and neither is visible from the score.

Worse, this gets more flattering as the system gets worse. A retrieval layer that misses a lot will have a small output list, which produces a small anchored relevant set, which produces good-looking recall against a bar you quietly lowered.

So: the relevant set is built blind. The labeller sees the query and the underlying memory store, decides what a correct answer contains, and only then is retrieval's output scored against it. Building it blind is more work and it produces worse-looking numbers, which is precisely the point — those numbers move when the system actually changes.

Rule two: contradiction is a gate, not a term

The second rule comes from thinking about what different errors cost the person using the product.

A missed fact is mostly invisible. The assistant doesn't mention where you live; you don't notice, because you have no idea what it could have said. The conversation is slightly less useful and nobody can tell.

A wrong fact is not invisible at all. The assistant confidently says you live somewhere you don't, or attributes to you a preference you never expressed. The user notices immediately, and what they learn is not "that fact was wrong" but "this thing makes things up about me." That's not a retrieval error any more, it's a trust failure, and it colours everything the assistant says afterward.

Those two costs are not on the same scale, so they should not be averaged together.

If contradiction is just a penalty term, a system can trade it away. Retrieve more aggressively, pick up more correct facts, pick up some contradicting ones too, and net out ahead on the aggregate score. The scoreboard improves. The product gets worse. That is exactly the kind of optimisation a metric will happily lead you into if you let it.

So contradiction isn't scored. Any contradicting fact fails the case outright, whatever the recall. A case that retrieves everything relevant and one thing that contradicts the record is a failed case, full stop.

Making it a gate rather than a weight means the system can never buy its way out. There's no recall number high enough to offset it, which is a fair description of how users actually feel about it.

Why this matters more for memory than for search

Both rules apply to retrieval generally, but memory raises the stakes.

Ordinary search results are proposals. The user sees ten links, picks one, and discards the rest — a bad result costs a moment of attention.

Retrieved memory isn't a proposal. It goes into the context and comes back out in the assistant's voice, as an assertion about the user, in the first person, with no visible provenance. The user has no way to tell which part of a sentence came from a retrieved fact and which the model composed. So a wrong retrieved fact is laundered into a confident claim, and the interface offers nothing to evaluate it against.

That asymmetry is the reason contradiction deserves a gate here specifically. In search, precision is a quality metric. In memory, it's closer to a safety one.

What generalises

Label blind, always. Any evaluation where the labeller sees the system's output before deciding what's correct will drift toward the output over time, and the drift is invisible in the score.

Not all errors are on the same scale, so don't average them. Work out which error the user actually punishes you for, and give it a different mechanism, not a bigger coefficient.

Gates beat weights for things you want to be non-negotiable. A weight is something an optimiser can trade away. A gate isn't.

Pick metrics by asking what a system could do to game them. If there's a way to improve the number while making the product worse, you've picked the wrong instrument — and the system will find that path whether or not anyone intends it to.