Launch is the moment a RAG system is most accurate. The index has just been built, the evaluation set has just passed, and the content owners have just checked their collections. From the next morning, every one of those things starts drifting.
Procedures are revised. Bids are submitted, won and lost. Someone restricts a folder. A new kind of project arrives with vocabulary the index has never seen. A document is withdrawn at the source and lives on in three places the pipeline made copies. None of this is dramatic, and none of it raises an alert unless someone designed one. The assistant simply becomes a little less right each week, and the users who notice first quietly stop trusting it.
This final part is about the operating work: who does what, how often, and how you know it is being done. It is the least glamorous part of RAG and the one that decides whether the investment in the other seven parts survives its first year.
Bergvliet Engineering is fictional, and its systems and figures are illustrative. Its Bid Library has been live for several months. The bid office and the four-person platform team agreed an operating model before launch, and then had to amend it after the incident described below.
Content changes are the main event
In most software, the code changes and the data follows. In RAG, the content is the product, and it changes constantly.
Re-index on change, not on a calendar. Bergvliet's pilot rebuilt the whole index weekly, refreshing only permissions nightly. That meant a revised procedure could be wrong in the assistant for up to six days, and a full rebuild took long enough to fail occasionally. The production pipeline listens for changes in the source libraries and re-indexes only the affected documents within the hour. A full rebuild still runs monthly, as a consistency check, and whenever the chunking specification or embedding model changes.
Version, do not overwrite. When a procedure moves from revision 3 to revision 4, the index replaces revision 3's chunks, but the evidence log (Part 3) keeps the fact that revision 3 was used for answers given before the change. If someone later asks why the assistant said something in June, the team can show which revision it was reading.
Watch for silent ingestion failures. A document that fails text extraction, or a library the connector lost access to, does not produce wrong answers; it produces missing ones. Bergvliet tracks indexed documents per collection against the source count and alerts when they diverge.
Deleting a document means deleting its copies
The incident that changed Bergvliet's operating model was small. Standard procedure SP-114, a temporary works method, was withdrawn after a safety review. The source library was updated within minutes. The Bid Library kept describing SP-114 as current for three weeks.
The investigation found three surviving copies. The search index had missed the deletion event because the connector treated "withdrawn" as a metadata change rather than a removal. A semantic answer cache was still serving an answer built from SP-114 to a common question. And an evaluation case still expected SP-114 as correct evidence, so the release gate would have failed a pipeline that stopped returning it.
The fix was to treat deletion as a process with a checklist, not an event:
- The source marks the document withdrawn and keeps the record, so history is preserved.
- Index entries, chunks and vectors are deleted the same day, and the deletion is verified by searching for the document's identifier.
- Cached answers that cited the document are invalidated.
- Evaluation cases that expected it are updated, with a new case asserting it is not returned.
- Logs keep the fact that it was used, and when, under the retention policy, but not a copy of its text.
The same checklist applies when a person asks for their personal information to be deleted under POPIA's data subject rights, which is a stronger reason to get it right. Deletion that reaches the source and not the copies is not deletion.
Every collection needs an owner
Ownership was a theme in Part 3; after launch it becomes a schedule. Every collection in the Bid Library has a named owner in the bid office or an engineering discipline, and once a quarter each owner confirms three things: the collection still belongs in the assistant, its authoritative documents are current, and its access rules are still right. A collection whose owner cannot be found, or will not confirm, is removed from the assistant's scope until someone takes it on.
This sounds bureaucratic. In practice it is a short meeting, and it is the only reliable defence against the slow accumulation of orphaned content that makes a knowledge assistant confidently out of date.
The operating calendar
Bergvliet wrote its recurring duties down as a calendar, with an owner for each line.
The weekly review of "not found" answers and negative feedback is the most useful hour in the calendar. It shows what people are asking that the corpus cannot answer, which is sometimes a retrieval fault and more often a gap in the content that the bid office can fill. It also shows query drift: new project types, new client names, new acronyms. When Bergvliet won its first large water treatment job, questions full of process terms the index had rarely seen began appearing within a fortnight. Those questions were added to the evaluation set, and two analyser synonyms were added so that lexical search connected the firm's internal terms with the client's.
Feedback is evidence, not a vote
The Bid Library has thumbs-up and thumbs-down buttons. They are useful, and they are easily misread. A thumbs-down might mean the answer was wrong, the evidence was missing, the user wanted a document rather than a summary, or the answer was right and unwelcome. Bergvliet asks one optional follow-up question after a thumbs-down ("What was wrong?" with four choices and a free-text box) and routes each kind differently: wrong answers to the diagnosis tree in Part 7, missing content to the collection owner, and format requests to the product backlog.
When it goes wrong
Treat the assistant like any other production system that people rely on. Bergvliet defined three kinds of RAG incident in advance:
- Leakage: restricted content reached someone who should not see it. The response is to disable the affected collection immediately, establish from the evidence log who received what, and follow the firm's information security incident process, including POPIA notification assessment where personal information is involved.
- Harmful staleness: the assistant gave current-seeming guidance from withdrawn or superseded content that someone could act on, as with SP-114. The response is the deletion checklist, plus a search of the logs for everyone who received the stale answer.
- Degradation: retrieval quality fell below the release threshold in production, for example after a connector change. The response is to roll back the change and rerun the evaluation set.
Each has a named first responder, and "disable the collection" is a single switch the bid office can use without waiting for the platform team. A kill switch nobody is allowed to press is decoration.
Keep cost per question visible
Cost drifts too. More users, longer questions, a reranker setting nudged upwards, a model upgrade that costs more per token: none of these is alarming alone. Bergvliet reports cost per question, split into retrieval, reranking and generation, on the same monthly page as recall and groundedness, so that a quality improvement and its price are discussed together. Part 6 covers the dials; the operating point is that someone looks at them on a schedule.
The whole series in one sentence
RAG that survives production is less about the cleverness of retrieval and more about discipline around it: search by meaning and by exact identity, follow the content's own structure, enforce the source's permissions at query time, send the model less and better evidence, measure retrieval before the answer, and run the whole thing with owners, calendars and deletion that reaches every copy. None of it requires a better model. All of it requires someone to own it.
Key points
Launch is peak accuracy. Content, permissions and questions start drifting the next day.
Re-index on change and version everything. Know which revision every past answer was reading.
Deletion must reach every copy. Index, chunks, caches and evaluation cases, verified, the same day.
Owners confirm their collections quarterly. Orphaned content is how assistants become confidently out of date.
Plan the incidents before they happen. Leakage, harmful staleness and degradation each need a first responder and a switch.
This is the final part. Start again at Part 1, Why vector search alone is not enough, see all parts on the series page, or read Memory is an architecture decision for the other half of what an AI system knows: what it should remember about the people it serves.
How CloudNala can help
We help organisations put an operating model around RAG systems that are already live: change-driven re-indexing, deletion that reaches every copy, collection ownership, an operating calendar, feedback routing, incident playbooks and a cost-and-quality report that lands on the right desk every month. It is the work that keeps a knowledge assistant trustworthy long after the launch team has moved on.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za