Retrieval-augmented generation is so common in AI tutorials that many people treat it as the definition of an AI project. Upload some documents, embed them, ask questions. It is usually the first thing a learner builds and the first thing a company asks for.
It is also frequently added for the wrong reason. Sometimes the knowledge in question is three paragraphs that could sit in the prompt. Sometimes it is structured data that should be looked up exactly, not searched for. And very often, once retrieval is in, nobody measures whether it is finding the right material, so every bad answer gets blamed on the model.
This part adds retrieval to the Umgeni assistant for one clearly stated reason and tests it in the order that matters.
The failure that justifies it
Umgeni Fibre is fictional, and its systems and figures are illustrative. After Part 4, the assistant remembers what the customer said and what they prefer. It still knows nothing about Umgeni. Asked "how do I reset my router?" or "what does the 100 Mbps package cost now?" it either declines or, worse, gives a plausible general answer that is wrong for Umgeni's equipment and prices.
The team first tried the obvious fix: put the facts in the prompt. That worked for the fair-use policy, which is short and changes once or twice a year. It did not work for the rest, for a reason that has nothing to do with size.
The real test: how fast does the knowledge change?
Umgeni's knowledge falls into four groups with very different clocks:
- Outage and maintenance notices, written by the network operations team, change by the hour. A planned maintenance window in Umbilo on Tuesday night is critical on Tuesday and wrong by Thursday.
- The price list and promotions change monthly.
- Router setup guides change when firmware or supported models change, a few times a year, and there are about 60 of them.
- The fair-use and billing policy changes rarely and fits on a page.
Nobody is going to edit and redeploy a prompt every time network operations posts a notice. The people who own that knowledge need to change it in their own tools, on their own schedule, and have the assistant reflect it. That is the actual case for retrieval: the knowledge must change independently of the system that uses it. Volume is a secondary reason. Plenty of small knowledge bases still need retrieval because they change often; plenty of large but static ones are better served another way.
The decision is short enough to draw.
What retrieval is not for
One question the team expected retrieval to answer was "is my line down?" It is not a retrieval question. The status of one customer's line is structured, current and specific, and it lives in the network management system. Searching documents for it would be slow and unreliable. The right design is an exact lookup through a tool, which is Part 6.
The general rule: if the answer is a record with an identifier, look it up. If the answer is in prose someone wrote, retrieve it. Many assistants need both, and confusing them is one of the commonest design errors in early AI projects.
Stage 3: a small, honest retrieval layer
Umgeni's first retrieval layer is deliberately plain. About 420 documents (notices, the price list, router guides and support articles) are split into passages, indexed for meaning with embeddings and also indexed for exact words. Each passage carries metadata: source, owner, published date, expiry date where one exists, and audience.
Three choices shaped it.
Keyword search alongside semantic search. Customers ask about router models by name and number. Semantic similarity is good at "my internet keeps dropping" and poor at an exact model code, where a keyword index is reliable. Hybrid search in plain language in the RAG series explains why many business systems need both.
Expiry as a filter, not a hope. Maintenance notices carry an end time. Expired notices are excluded at query time, so the assistant cannot tell a customer about last week's maintenance as if it were this week's.
Audience as a separate index. Umgeni has internal documents too: technician escalation guides and the commercial terms for business customers. Rather than filter them out of a shared index, the customer assistant has its own index containing only customer-facing material. A filter can be misconfigured; a document that was never indexed cannot leak. Permission-aware retrieval covers the harder case, where different users are entitled to different documents in the same index.
The prompt tells the model to answer only from the passages provided, to cite them, and to say plainly when the passages do not answer the question.
Test what it finds before what it writes
This is the lab's most important habit. The team wrote 80 test questions, each paired with the document that should answer it. Some were simple, some used router model codes, some were in isiZulu or Afrikaans, a few were about expired maintenance and a handful had no answer in the knowledge base at all.
They measured retrieval first, with no model in the loop: for each question, was the correct document among the top five passages returned? In the first version the answer was yes for 71% of questions (illustrative). The misses were instructive. Questions with model codes failed on semantic search alone; two expired notices were still being returned because their expiry dates had been typed into the body text rather than the metadata field; and questions in isiZulu performed worse than the same questions in English.
Fixing those (adding keyword search, correcting the metadata and adding isiZulu phrasings to the test set and to two key articles) raised the figure to 92%. Only then did the team evaluate the answers: whether each was supported by the retrieved passages, whether citations pointed to the passages they claimed, and whether the assistant said "I don't have that information" for the questions with no answer.
Doing it in that order matters. If the right document is not retrieved, no model can produce a grounded answer, and tuning the prompt is wasted effort. Evaluate retrieval separately from the answer goes into the metrics properly.
One question through the layer
A worked example shows how the pieces fit. A customer in Umbilo writes, partly in isiZulu: is there maintenance tonight, and will my ZX-300 router need resetting afterwards?
The question carries two needs. The maintenance part matches a notice network operations posted that morning, found by meaning ("maintenance tonight") and kept because its end time is in the future. The router part is found by the exact model code, which the keyword index matches to the ZX-300 guide; semantic search alone ranked a different router's guide higher. Both passages go to the model with their source and date.
The answer cites both: maintenance from 22:00 to 02:00, and a note that the ZX-300 reconnects on its own but may need a restart if the light stays red after 02:30. If the customer then asks whether their own line is affected, that is no longer a retrieval question. It is a lookup of their line, and the assistant says it will need a tool to check, which is exactly the gap Part 6 fills.
What the stage makes worse
Retrieval introduces an operating burden that demos never show. Someone must own each document source and keep it current. The index must be refreshed when content changes: Umgeni refreshes notices every 15 minutes and everything else nightly. Deleted documents must disappear from the index, not just from the website. And the evaluation set needs new questions whenever new kinds of content arrive.
It also widens what the model sees, so cost per answer rises; in Umgeni's illustrative figures, about threefold against the stage 2 assistant, still a few cents per conversation. And it still cannot do anything for the customer. Someone whose line is down wants a technician, not an article. That is the next stage's reason.
Key points
Retrieval is justified by independent change. If the knowledge's owners must update it on their own schedule, retrieve it.
Records are looked up, prose is retrieved. A customer's line status is a tool call, not a search.
Prefer separate indexes to filters for audience. What was never indexed cannot leak.
Measure retrieval with no model in the loop. Was the right document in the top results? Fix that first.
Retrieval is an operating commitment. Owners, refresh schedules, deletion and an evaluation set that grows with the content.
Next: Part 6, Add tools only when the system must act. All parts are on the series page.
How CloudNala can help
We help teams decide whether retrieval is actually the right tool, design the indexes, metadata and refresh schedules around the people who own the knowledge, and build an evaluation set that tests what is found before what is written. That turns "the AI gives bad answers" into a specific, fixable list.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Plan an AI MVP Build Sprint or write to us at consult@cloudnala.co.za