A person who remembers something usually also knows how old the memory is, where it came from and whether it still applies. "I think she lived in Port Elizabeth, but that was years ago and the city has been renamed since." That surrounding judgement is most of what makes human recall useful.
An agent's memory system does not come with that judgement. When it retrieves stored items, it returns whatever is most similar to the current question. Similar is not the same as current, permitted, relevant or true. The retrieved items are candidates. Whether the agent may use them is a separate decision, and it is the one teams most often leave out.
Part 4 covered what is allowed into memory. This part covers what is allowed out.
A retrieval that went slightly wrong
Sunbird Home is fictional, and its systems and figures are illustrative. Zanele Mokoena writes to Sunbird Help on WhatsApp: "Can you send the table to my office instead? I won't be home on Friday."
In an early version of Sunbird's assistant, the retrieval step searched her preferences and earlier case records for anything similar, and returned twelve items. Among them: two staff-only notes from a previous case about a payment query; a delivery note asking for "delivery after 3pm at home" from March, attached to an address she no longer uses; three items about a different order, a set of chairs; and her stored preference for English, set in an earlier conversation, while this message was in Afrikaans.
The early assistant used most of what it was given. It quoted the staff notes on the payment query, which she had never seen, offered a 3pm slot at the old address and replied in English. Every item was genuinely about Zanele. None of them should have been used as they were.
Four checks between retrieval and use
Sunbird now passes every retrieved memory through four checks before the assistant may use it.
Permission. Is the person or process asking entitled to see this item? Staff-only notes exist in case records for good reasons, and the assistant acts on the customer's behalf, not as a supervisor. Permission must be checked against the access rules that apply now, for this requester, not the rules that applied when the item was written. Two items were dropped here.
Freshness. Is the item still likely to be true? Every stored item at Sunbird carries a written date, a source and a review date. The 3pm note referred to an address that the order system no longer lists for her, so it no longer described anything. Four items were dropped, all tied to the old address.
Relevance. Does it bear on this request? The chairs were a different order. Similarity search returns them because they look alike ("delivery", "late", "Zanele") but they answer a different question. Three were dropped.
Conflict. Does it contradict something more authoritative? Her stored preference said English. Her current message is in Afrikaans. The current conversation wins. One preference was overridden, and the assistant now asks before changing the stored preference rather than silently keeping either.
Two items survived: the open courier query for the table, and her preference for WhatsApp updates. The reply could also say where each came from, which matters for the next section.
These checks are not exotic. They are the same questions a careful person asks: am I allowed to say this, is it still true, is it about this, and does something newer contradict it? The difference is that a system will only ask them if someone builds the checks.
Build the checks in code, not in the prompt
The tempting shortcut is to hand the model all twelve items with an instruction: "ignore staff-only notes and anything out of date". That moves the check into the one component that cannot be relied on to apply it consistently. The model may follow the instruction most of the time. It has still seen the staff notes, and a cleverly phrased customer message can coax them out.
Sunbird runs permission and freshness as deterministic filters before anything reaches the model. Items the requester may not see are never retrieved for that requester in the first place, by applying the access rules inside the search query rather than afterwards. Expired items are excluded by date. In the diagram above, that means the first six drops now happen before the model sees anything. Relevance and conflict are partly judgement, so the model helps there, but within rules the application enforces: an item about a different order number is removed by matching identifiers, and conflicts are resolved by the written precedence order below rather than by the model's preference. The principle is simple. If a check protects someone, it belongs in code.
When sources disagree
Conflict is the hardest of the four, because it needs a rule about which source wins. Sunbird wrote that rule down.
What the customer says now comes first, for anything about her own wishes. If she says "send it to my office this time", that is the instruction, whatever the stored preferences say.
The system of record comes next, for facts: order status, payments, courier status, addresses. If memory says one thing and the order system says another, the order system is right and the memory is a defect to be corrected.
The current policy version governs how the case is handled. A memory that "last time, the supervisor waived the delivery fee" is a record of what happened, not a precedent the assistant may apply. Policy version 4 decides.
Validated memory, with its source is fourth. It is useful for defaults and continuity, and it never overrides the three above.
An inference is last. At most it prompts the assistant to ask a question.
Most incidents with agent memory are cases where this order was inverted: a remembered price beat the price list, a remembered entitlement beat policy, or a remembered address beat the order system.
Freshness needs dates, and dates need to be written
The freshness check only works if items carry the information it needs. That goes back to the write policy. An item without a written date, a source and a review date cannot be checked for freshness, and the safe treatment is to ignore it.
Sunbird uses three simple rules:
- Items derived from another system expire when the source changes. The 3pm note was tied to an address record. When that record changed, the note should have been flagged for review automatically.
- Preferences have review dates. Twelve months, after which the assistant asks again rather than assuming.
- Nothing is fresh because it was recently retrieved. Being used often is not evidence of being true.
Show where a memory came from
The last practical step is to let the reply carry provenance where it helps. When Sunbird Help says "I'll send the update on WhatsApp, as you asked on 2 September", three things happen. Zanele can see what the assistant remembers. She can correct it if it is wrong. And a reviewer reading the trace later can tell that the memory was used and where it came from.
This also exposes invented memories quickly. An assistant that says "as you mentioned last time" when nothing was stored is making something up. When provenance is required, it has nothing to cite, and that missing citation is a signal the evaluation in Part 7 can test for.
The overlap with retrieval over documents
Everything here also applies to retrieval over documents, which is what most people mean by RAG. Permission, freshness, relevance and conflict are the same four questions whether the candidate is a customer preference or a paragraph from a policy manual. The companion series RAG That Survives Production covers the document side in depth, including permission-aware retrieval, and CloudNala's earlier article on why RAG is a data security system makes the security case.
The order of the two series matters. A team first needs to know what the system should retain and why, and only then how it retrieves knowledge that changes outside it.
Key points
Retrieved means candidate. Similarity does not establish permission, currency or truth.
Four checks before use. Permission, freshness, relevance and conflict, every time.
Write the precedence order down. Customer now, system of record, current policy, validated memory, inference.
Freshness depends on the write path. Items with no date or source cannot be checked, so ignore them.
Show provenance. It lets people correct memory, and it exposes invented recall.
Next: Part 6, Agent memory, POPIA and South African organisations. All parts are on the series page.
How CloudNala can help
We help teams put the checks between retrieval and use: permission against current access rules, freshness from source and dates, relevance beyond similarity, and a written precedence order for conflicts. We test them with real retrieval traces from the system, looking for the cases where the agent used something it found but should not have.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agent Memory Review or write to us at consult@cloudnala.co.za