There is a moment in most assistant projects when someone proposes making everything searchable. The help pages are already in a search index for the assistant. Why not add statements, transaction histories and call notes to the same index, tag each with the client it belongs to, and filter on the client at query time? One pipeline, one store, one retrieval step, and the assistant can answer almost anything.
It is a reasonable engineering instinct, and for a bank it is the wrong architecture. Chapter 4 explains why Impala Bank drew a hard boundary between public knowledge and private account data instead, and what that boundary had to include to be real rather than a diagram.
Where we are in the case study
Impala Bank is fictional, and its systems and figures are illustrative. By the end of Chapter 3, Ask Impala could answer public questions from approved content and, inside the signed-in app, explain a declined card or a debit order using a narrow, masked slice of account context fetched by the bank's own account service. The model never chose whose data to fetch.
The proposal to "index everything" arrived when the team wanted the assistant to answer questions such as "when did I last pay my municipal account?", which meant searching a client's transaction history by meaning rather than by exact fields.
One filter between a stranger and a statement
In the single-index design, the only thing that stops one client's statement appearing in another client's answer is a filter applied at query time. If that filter is correct, on every query path, forever, the design works.
The problem is everything that has to stay true for that to hold. Every new query path must apply the filter. Every re-indexing job must carry the client tag correctly. Every vendor feature, such as a new semantic ranking option or a cached results layer, must respect it. Every engineer debugging a retrieval problem must not run an unfiltered query against production data. And every test must cover the case where the filter is absent, which is the case nobody remembers to write.
When the boundary between public and private is a filter, a single mistake turns a help-page search into a data breach. When the boundary is architecture, the same mistake produces a help-page search that returns help pages. That asymmetry is the whole argument.
Our earlier article RAG is a data security system makes the general case that retrieval decides who sees what. This chapter is the specific version for a bank assistant.
What the boundary actually contains
Impala's design kept two worlds apart.
Outside the boundary: public knowledge. The approved help pages, fee tables and product terms from Chapter 2. They are the same for every client, anyone may query them, and their index holds no personal information. A mistake here can produce a wrong answer, but not a disclosure.
Inside the boundary: account data. Statements, transactions, debit orders, card status and anything else tied to a client. This data is never copied into the assistant's search index. It stays in the systems that already own it and is reached only through account services that check the session on every call, as in Chapter 3.
The boundary is not just two databases. To be real it had to cover five things.
- Storage. No client data in the public index, ever. A scheduled scan checks the public index for anything resembling account numbers, ID numbers or names linked to balances.
- Retrieval. Searching a client's own history by meaning is done by the account service, inside the boundary, over that client's records only. The assistant asks the service a question; it does not search a shared store.
- Logging and traces. Conversation logs that include account context are classified and stored separately from logs of public conversations, with tighter access and shorter retention.
- Retention. Public-conversation logs are kept long enough to improve content. Signed-in conversation logs follow the bank's records rules for client interactions, decided by its records and privacy teams rather than by the AI team's convenience.
- People. Content designers improving public answers can read public transcripts. They cannot read signed-in transcripts without a specific, logged reason.
Searching a client's own history
The "when did I last pay my municipal account?" question still needed answering. Impala's answer was to do the semantic part inside the boundary, scoped from the start.
The account service received the question and the session. It fetched that client's transactions for a sensible window, typically the last twelve months, and ran the matching over that set alone. There was no shared index of all clients' transactions to filter, so there was no filter to get wrong. The service returned the matching transactions, masked as usual, and the assistant phrased the answer.
This is slower and more expensive per query than a pre-built index over all clients. It is also the design where a bug cannot reach another client's data, because that data is never in the set being searched. For a bank, that trade is easy.
Logs are part of the boundary
Logs are where boundaries usually leak. An assistant team wants transcripts to improve the product, and a developer debugging a bad answer wants to see exactly what context the model received. Both are legitimate. Both mean copies of account data accumulating in tools that were never designed to hold it.
Impala made three rules. Signed-in conversation traces mask the same fields the account service masks, before they are written. Access to unmasked operational detail, when an investigation genuinely needs it, goes through the bank's existing privileged-access process with a ticket and an expiry. And evaluation datasets, the sample conversations used to test new versions, are built from public conversations or from synthetic clients, never from copied production account data.
That last rule cost the evaluation team some realism, which they recovered by building synthetic clients carefully: varied account types, unusual transaction descriptions, multiple languages and the awkward edge cases real clients produce.
POPIA as a design question
None of this is presented as a legal conclusion. It is general information, not legal advice, and Impala's privacy office and legal advisers made the determinations. But several of POPIA's conditions for lawful processing map directly onto the boundary as design questions.
Minimality: is the personal information adequate, relevant and not excessive for the purpose? Keeping account data out of a general search index answers that more convincingly than any filter. Retention: records should not be kept longer than necessary for the purpose, subject to legal exceptions, which is why signed-in logs have their own retention rule. Security safeguards: the responsible party must take appropriate, reasonable technical and organisational measures, which is what the separation of storage, logs and access is. If any processing happens offshore, such as at a hosted model provider, section 72's conditions on cross-border transfers become part of the picture. Our plain-language explainer POPIA does not require your data to stay here covers that point, and Chapter 8 returns to it.
Testing the boundary
The tests for Chapter 4 were boundary tests, run continuously rather than once. Search the public index for patterns that look like personal information. Confirm that the public retrieval path has no credentials that could reach account systems at all, so even a misconfigured query cannot cross. Attempt cross-client queries against the account service. Check that signed-in traces are masked before storage. Check that the evaluation datasets contain no production account data.
The test that mattered most was the second one. A boundary is strongest when the public side does not merely choose not to cross it, but has no means to.
Key points
A filter is not a boundary. One shared index with client tags turns a single mistake into a disclosure.
Keep account data where it lives. Reach it through session-checked services; never copy it into the assistant's search index.
Search a client's history inside the boundary. Scope the set to one client first, so there is no filter to get wrong.
Logs, retention and people are part of the boundary. Mask before writing, separate signed-in traces, and control who reads them.
Make crossing impossible, not just forbidden. The public path should hold no credentials that can reach account systems.
Next: Chapter 5, Read-only tools: letting the assistant look, not touch. All chapters are on the series page.
How CloudNala can help
We help organisations design the boundary between public knowledge and private customer data in AI systems: what is indexed where, how customer-specific search is scoped, how logs and evaluation data are protected, and how the boundary is tested continuously. The result is an architecture your privacy and risk teams can inspect, not a filter they have to trust.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Production Readiness Review or write to us at consult@cloudnala.co.za