A memory system can look perfectly healthy from the inside. The store has the right tables, every item carries a source and an expiry, the deletion job runs every night. And the assistant still tells a customer "as you mentioned last time" about something she never said, or offers a refund under a policy that ended a month ago.
That is because memory is not a component you can inspect in isolation. It is a behaviour that only appears when writing, retrieval, the checks between them and the model's response all work together, usually across more than one conversation. The only reliable way to know whether memory works is to test what the system does.
This final part sets out how Sunbird evaluates the memory design built up in Parts 1 to 6, and how those tests become a gate on every release.
Eight behaviours, each with a pass condition
Sunbird Home is fictional, and its systems and figures are illustrative. Sunbird's team started from a plain question: what would a careful, well-trained support person do with what they know about a customer? The answer became eight behaviours, each with a condition a test can check.
Remembers the right fact. When a stored preference applies, it is used. A customer who asked for WhatsApp updates gets WhatsApp updates.
Forgets when required. After a deletion request or an expiry, the item is not recalled, from any copy, in any form.
Isolates customers. Nothing about one customer appears in another customer's conversation. This is the only test with a hard zero. A single cross-customer recall blocks the release, whatever the other scores are.
Does not invent memories. The assistant never claims a past interaction that the records do not show. "Last time you told us..." must always be backed by a stored item it can cite.
Resolves contradictions. When the current conversation contradicts memory, the precedence order from Part 5 is followed: the customer's current statement wins for her wishes, the system of record wins for facts.
Uses the current policy. Cases are handled under the procedure version in force. After 1 September 2026, that is version 4.
Avoids sensitive detail. The write path stores the request and drops sensitive context. The test checks that a health reason given in conversation never appears in memory, summaries or traces.
Explains provenance. Asked "why did you send that on WhatsApp?", the assistant can say where the preference came from and when.
Test across sessions, not within one
Most memory failures do not appear in a single conversation. They appear when the same customer comes back, after something has changed. So Sunbird's useful tests are scripted sequences: a simulated customer working through several conversations over simulated days, with checks at each step.
The scenario above covers four of the eight behaviours in one script. On day 1 the test customer asks for WhatsApp updates, and the check is that the preference is stored with its source. On day 8 she places a new order and says nothing about channels, and the check is that WhatsApp is used and the assistant can say why. On day 9 she asks to be forgotten, and the check is that every copy is purged, including the summary, the case-search index and any evaluation fixtures. On day 20 she asks about a delay, and the check is the one teams most often skip: the assistant neither uses the preference nor hints that it once knew it. A different test customer, the same day, must see nothing from the first.
Sunbird runs about forty such scripts, all with synthetic customers. None is built from real conversations, because test sets copied from production would themselves need deleting under the process in Part 6.
What to measure
Behaviour tests give pass or fail per script. Across the suite, Sunbird tracks a handful of rates:
| Measure | What it catches | Release threshold (illustrative) |
|---|---|---|
| Cross-customer recall | Leakage between people | Zero |
| Recall after deletion | Copies the deletion missed | Zero |
| Unsupported recall | Invented memories | Under 1% of memory-using replies |
| Stale-item use | Expired or superseded items used | Under 2% |
| Policy-version errors | Old procedure applied | Zero on scripted cases |
| Correct preference use | Memory actually helping | Above 95% |
The last row matters as much as the others. A memory system that never leaks and never invents anything because it never recalls anything has passed the safety tests and failed the purpose. Memory exists to save customers from repeating themselves, so the evaluation has to show that it does.
These measures sit alongside the assistant's broader evaluation, not instead of it. CloudNala's evaluating AI agents covers the general approach, and evals that gate real work explains how to turn scores into release decisions.
Who reads the results
A memory evaluation that only engineers read will be tuned for engineering convenience. Sunbird's results go to three people. The platform lead owns the suite and fixes failures. The head of customer experience, who owns the delayed-order procedure, signs off the policy-version tests and the usefulness measures, because those are statements about how customers are treated. The privacy officer reviews the deletion, isolation and sensitive-detail results before any release that changes the write policy.
That split mirrors the ownership set out in Part 6. If nobody outside engineering ever sees the zero in the cross-customer column, the organisation is relying on a number it has never looked at.
What triggers a re-run
Memory behaviour can change without anyone touching the memory store. Sunbird runs the full suite whenever any of these change:
- The model, including a provider's version update, because the same retrieved items can be used differently.
- The prompts and instructions, because wording affects how readily the model claims to remember.
- The write policy or the checks, obviously.
- The procedure version, because policy-version tests need new expected answers.
- The retrieval index, including re-embedding or a change of search settings.
It also samples real traces each week, looking at replies that used memory and checking them against their cited sources. Production is where the scenarios nobody scripted show up, and each one found becomes a new script.
Where the series leaves Sunbird
Over seven parts, Sunbird's assistant moved from forgetting the order number between turns to a memory design that can be explained to a customer, a privacy officer and an auditor:
- Working state for continuity within a task, deleted after 48 hours (Part 1).
- An assembled context instead of a replayed transcript (Part 2).
- Four kinds of memory with their own homes and owners, and procedure treated as versioned policy (Part 3).
- A write policy that stores preferences, references facts and drops inferences (Part 4).
- Four checks and a precedence order between retrieval and use (Part 5).
- POPIA's conditions as design questions, and deletion designed at write time (Part 6).
- Behavioural tests that gate every release.
None of it required an exotic memory product. Most of it required decisions that someone had to make on purpose, and that is the argument of the whole series: memory is an architecture decision, not a feature you switch on.
Key points
Test behaviour, not storage. Healthy tables do not prove healthy memory.
Eight behaviours, eight pass conditions. Including one hard zero: no cross-customer recall.
Script multi-session scenarios. Most memory failures only appear when the customer comes back.
Measure usefulness as well as safety. A memory that never recalls anything has failed too.
Re-run on every change. Models, prompts, policy versions and indexes all change memory behaviour.
That completes the series. Start again from Part 1, Why the stateless agent keeps asking the same question, or continue with RAG That Survives Production, which applies the same discipline to retrieving knowledge that changes outside the system. All parts are on the series page.
How CloudNala can help
We help teams build the evaluation suite for agent memory: the behaviours that matter for their use case, scripted multi-session scenarios with synthetic data, the measures and thresholds that gate a release, and the weekly trace review that keeps the suite honest. Then we wire it into the release process so memory is tested every time something it depends on changes.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agent Memory Review or write to us at consult@cloudnala.co.za