Here is a bug that ships in almost every popular AI-memory layer, and almost nobody talks about it. Your user asks a question, and your memory store writes the question down as if it were a fact. The next time the model recalls, it confidently tells the user something they never said. It just inferred it from their own question.
This is not a rare edge case. It is the default behavior of the architecture that mem0, Zep/Graphiti, Letta, Cognee, and most home-grown RAG memory pipelines share. The pattern has a name worth knowing, because once you can name it you can fix it: extract-then-store.
The extract-then-store trap
The standard memory loop looks like this. A turn comes in. You run an extractor over it (often an LLM prompted to "pull out facts about the user"). Whatever the extractor returns gets embedded and written to the store. On recall, you do a similarity search and feed the hits back to the model.
The flaw is in the second step. The extractor is asked what facts are here, never is this even an assertion the user is making about themselves. So it happily turns interrogatives into declaratives:
user: "What stage of my cycle am I at?"
extractor output: "user's cycle is at stage X"
stored fact: "user's cycle is at stage X" // WRONG. It was a question.The user asked. The system answered itself, then believed its own answer. This exact failure corrupted a health record in a personal-assistant project we run, which is what sent us looking for the general fix.
Four ways non-facts leak in
Once you start auditing real traffic, you find that questions are only one of several speech acts that should never become facts. Here are the four that leak most often:
- Questions. "Where do I live?" is a request for recall, not an assertion. Storing it manufactures a fact out of thin air.
- Hypotheticals and wishes. "I wish I lived by the sea" is a desire, not a residence. Stored as fact, it relocates the user.
- Commands. "Remind me to call the dentist" is a task, not a biographical fact. It belongs in a todo system, not a belief store.
- Third-party claims. "My wife prefers tea" is a fact about someone else. Conflating it with a fact about the speaker is one of the most common memory-corruption bugs in the wild.
Each of these is grammatically adjacent to a real fact. "I live in Bucharest" and "Where do I live?" share almost every token. An extractor optimized for recall will treat them the same. That is the whole problem in one sentence: extractors are tuned to find facts, not to refuse non-facts.
Why "just prompt the extractor better" does not hold
The obvious reflex is to add "only extract genuine first-person assertions" to the extractor prompt. This helps a little and fails in production for three reasons.
First, it is non-deterministic. The same utterance can pass on Monday and fail on Tuesday, so you cannot write a regression test that stays green. Second, it is expensive. You are now paying for a second reasoning pass on every single turn, most of which are mundane. Third, it is invisible. When a bad fact does leak, there is no audit trail that says which rule decided this was storable, so you cannot debug it. You just have a confidently wrong assistant and no log to point at.
What you actually want is a cheap, deterministic gate that runs before the store, classifies the speech act of the user's raw message, and only forwards genuine assertions. The expensive LLM pass becomes a fallback for the residual hard cases, not the primary filter.
Classify the speech act, then decide
Linguistics gave us the vocabulary for this decades ago. A speech act is what an utterance does: assert, ask, request, wish, deny. Memory hygiene reduces to one rule: only assertions about the speaker become facts. Everything else is logged for context but never promoted to a belief.
This is exactly the job of the open-source speech-act-memory-gate we maintain. It is a tiny, zero-dependency module you put in front of whatever store you already use. You feed it the user's raw message and it returns a decision:
import { gate } from "speech-act-memory-gate";
gate("I'm allergic to hazelnuts.").action; // "store"
gate("Am I allergic to hazelnuts?").action; // "drop" (question)
gate("Remind me to buy almonds.").action; // "drop" (command)
gate("My wife is allergic to shellfish.").action; // "drop" (third-party)
gate("I no longer drink coffee.").action; // "reinforce" (contradiction)The three actions map cleanly onto a real store. store writes a new belief. drop never touches the store. reinforce is the interesting one: an explicit denial or correction should not create a new row, it should update the confidence of an existing belief. We will come back to that asymmetry in the next post on the trust model.
Wiring it in front of mem0
You do not throw out your existing store. The gate sits in front of it. The whole integration is a single conditional:
import { gate } from "speech-act-memory-gate";
import { MemoryClient } from "mem0ai";
const mem0 = new MemoryClient({ apiKey: process.env.MEM0_KEY });
async function ingest(userMsg: string, userId: string) {
const d = gate(userMsg); // classify the USER's message only
if (d.action === "drop") return; // questions/commands/3rd-party never stored
if (d.action === "store") {
await mem0.add([{ role: "user", content: userMsg }], { user_id: userId });
}
// d.action === "reinforce": look up the matching fact and update its trust
}One detail matters more than the rest: pass the user's raw message, never the assistant's reply, and never the two concatenated. Classifying the assistant's explanation as if the user asserted it re-creates the original bug from a new direction. The gate decides about what the human said, full stop.
How to measure whether it worked
The metric that matters is precision: of everything you stored, how much was actually a fact. Recall (did you catch every real fact) is easy to max out; you can store everything and hit 100% recall while your store fills with garbage. Precision is the number that protects your user from confident lies.
On a small curated benchmark of 60 labeled utterances across English, Hebrew, Russian, and Romanian, an extract-then-store baseline scored 35.2% precision: it stored 35 non-facts out of 54 writes. The same set through the gate scored 100% precision with no loss of recall. The set is deliberately adversarial and deliberately small, so read it as "no known regressions on a realistic hard set," then extend it with your own traffic and run it in CI. The harness exits non-zero on regression, which is the point of having it.
The takeaway
If your agent has ever told a user something they never said, check your ingest path before you blame the model. The model is usually fine. The store fed it a fact that was never a fact, manufactured out of a question the user asked weeks ago. Add a speech-act gate in front of the store, measure precision, and the whole class of bug disappears. It is the cheapest reliability win available in agent memory today, and it is one conditional.