The second proposal for Bontebok Furniture Group's delivery problem arrived a week after the first had been parked. Bontebok is a fictional South African furniture retailer with stores and home delivery, and its people, systems and figures in this series are illustrative. The proposal came from a capable developer in the digital team and it was more sophisticated than a plain chatbot. It would index the shared deliveries mailbox, the distribution centres' email threads and the delivery policy PDF into a vector store, and let a model answer customers with retrieval-augmented generation, usually shortened to RAG.
It was a good demonstration. Asked "where is order 4471-882?", it found an email from a dispatcher two weeks earlier that mentioned the order and answered with confidence. The order had been rescheduled twice since then. The email was the most similar text in the index. It was not the truth.
Part 1 showed that most of Bontebok's delivery calls came from information that customers and staff could not see. This part is about what happens when a team tries to retrieve its way around that gap, and what to fix first.
Retrieval finds text. It does not create truth.
RAG is a pattern in which a system searches a collection of content, puts the most relevant pieces into the model's context and asks the model to answer from them. Our explainer, RAG explained simply, covers the mechanics, and the RAG That Survives Production series covers how to build the pipeline well.
Everything in that pipeline assumes one thing: that a correct answer already exists somewhere in the collection. Retrieval can rank, filter and cite. It cannot make up for an answer that was never written, was written once and then went out of date, or lives in a system the index cannot see. When the answer is not there, a well-built pipeline returns the closest thing it can find, and a fluent model turns that into a confident sentence.
That is why the order of work matters. Close the information gap first. Then decide whether retrieval is the right way to reach what you now have.
Five kinds of information gap
When Riaan, Bontebok's solution architect, went through the delivery questions one by one, every failure fell into one of five kinds of gap. Naming them changed the conversation, because each has a different fix and only one of them is helped by retrieval.
Missing. The information was never captured. Bontebok recorded the date a customer was promised at the till, but not the reason a delivery was later moved. No index can retrieve a reason nobody wrote down. The fix is to capture it at the point where it happens, usually as a required field or an event.
Trapped. The information exists in a system, but not where the people or channels that need it can reach it. Pick confirmations sat in the warehouse system. Truck assignments sat in the route planner. The fix is an integration or a shared service, not a search engine.
Stale. The information was copied once and never updated. The promised date on the order record was calculated at the till from a fixed lead-time table and never changed, even when the warehouse knew the stock would be late. Indexing stale data makes it easier to find, which makes things worse.
Contradictory. Two sources disagree. For outlying towns the carrier's portal and Bontebok's route planner sometimes showed different delivery days. Retrieval will happily return whichever is more similar to the question. The fix is to decide which source is authoritative for each fact, and to record that decision.
Ownerless. The information exists but nobody maintains it. Bontebok's delivery policy PDF still described a fee that had been changed eight months earlier. Assembly guides were scattered across supplier websites. Retrieval over ownerless content is retrieval over decay.
Only after a gap has been closed does the question "should we use RAG to reach this?" become sensible. This is the practical edge of the argument in No AI without information architecture: the rung that makes answers trustworthy has to be built before the rung that generates them. And for some information, the answer is still no.
Operational state is not a retrieval problem
The most important distinction in Bontebok's audit was between two kinds of question.
Some questions ask about knowledge: what is the delivery fee to Knysna, how do I care for a leather couch, what happens if I am not home. The answer is the same for every customer, it changes occasionally, and it lives naturally in documents. Retrieval can be a good fit for knowledge, especially when there is a lot of it and it changes faster than anyone would retrain a model. Our article on adding retrieval only when knowledge must change sets out that test.
Other questions ask about state: where is my order, when will it arrive, why did only the base of the bed come. The answer is different for every customer, it changes by the hour, and it has exactly one correct value at any moment. State belongs in a system of record and should be read with a precise query, by order number, for an authenticated customer. Searching a pile of emails for text that resembles the question is the wrong tool even when the emails happen to be accurate.
Bontebok's demonstration failed because it treated state as knowledge. Most of the delivery demand from part 1 was about state. A correct design reads state from the delivery status service described in part 1 and reserves retrieval for the much smaller set of knowledge questions. The same separation appears in the Production AI Case Study, where a bank keeps public knowledge and private account data on different paths.
The mailbox was also a privacy problem
The proposal had a second weakness that had nothing to do with accuracy. The deliveries mailbox held customers' names, home addresses, cellphone numbers, gate codes and, occasionally, notes about who would be home and when. Indexing it would have created a new, searchable copy of that personal information for a new purpose, available to a model answering anyone who asked about any order.
The Protection of Personal Information Act, 2013 (POPIA), section 15, requires further processing of personal information to be compatible with the purpose for which it was collected. Whether a particular use meets that test is a question for Bontebok's information officer and its legal advisers, and this article is general information, not legal advice. Section 19 also requires a responsible party to secure the integrity and confidentiality of personal information with appropriate, reasonable technical and organisational measures, and a new searchable copy of gate codes and home addresses is exactly the kind of asset those measures must cover. The architectural point is simpler: an index built from a shared inbox inherits every access problem the inbox had, and adds new ones. Our article on why RAG is a data security system covers the controls that retrieval over personal information needs, and permission-aware retrieval covers how to enforce them.
Reading a single order's status through an authenticated query does not have this problem in the same way. The customer sees their own order and nothing else.
An information gap audit
Before Bontebok allowed anyone to build an index, Riaan ran a short audit over the top question types from the demand analysis. It takes a few days with the right people in the room, and it is reusable for any request to put AI in front of customers or staff.
| For each question type | Ask | Bontebok example: "When will my bed arrive?" |
|---|---|---|
| State or knowledge? | Is the answer specific to one case and changing? | State |
| Where is the truth? | Which system holds the authoritative value? | Route planner once scheduled; warehouse before that |
| Can staff see it today? | Could an agent or store answer correctly now? | No |
| Which gap? | Missing, trapped, stale, contradictory or ownerless? | Trapped and stale |
| Who owns it? | Who keeps the value correct and fixes errors? | Logistics, per distribution centre |
| Freshness needed? | How quickly must a change reach the customer? | Same day for date changes |
| Personal information? | Does answering expose anyone's data? | Yes; authenticated access only |
| Fix before AI | What closes the gap? | Status service and date-change event |
The audit produced a short list of fixes that had nothing to do with AI: an event when the expected date moves, an authoritative-source rule for carrier deliveries, a named owner for the delivery policy page and a single place for assembly guides. It also produced a much smaller and better-defined role for retrieval than the original proposal.
Where RAG did end up
Retrieval earned a place at Bontebok, but not where it was first proposed. Once the delivery policy, fee table and product care guides had an owner and a review date, and once they were published in one place, they became a sensible collection for a retrieval layer. Customers asking "can you deliver to a third-floor flat with no lift?" or "how do I clean a fabric couch?" get answers from approved content, with a link to the source page.
Questions about a specific order never touch that index. They go to the status service, which answers from the current record. Part 4 shows how the two paths meet behind one conversational front door.
The demonstration that started this part was not wasted. It showed, more clearly than any slide could have, what happens when a capable model is pointed at an information gap. That is a useful thing to see before launch rather than after.
Key points
- Retrieval can only return information that exists, is current and has an owner. Close the gap first.
- Name the gap: missing, trapped, stale, contradictory or ownerless. Each has a different fix, and only some are helped by retrieval.
- Separate state from knowledge. Read state from the system of record with a precise, authenticated query; reserve retrieval for maintained knowledge.
- Do not index shared mailboxes or case notes to work around a visibility problem. It creates a privacy and accuracy problem as well.
- Run an information gap audit over the top question types before anyone builds an index.
Next: Part 3, Process mining before agent design. All parts are on the series page.
How CloudNala can help
CloudNala runs information gap audits for teams planning a RAG system or AI assistant. We take your top question types, trace each answer to its source of truth, classify the gaps, separate state from knowledge and identify the fixes that should come before any index is built, so that retrieval is used where it helps and is not asked to cover for information your organisation has not yet captured or maintained.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Start a Digital Discovery Sprint or write to us at consult@cloudnala.co.za