When a RAG system gives a bad answer, the conversation usually turns to the model. Is it hallucinating? Would a newer one be better? Should we tune the prompt?
Sometimes that is the right conversation. More often the answer was decided several steps earlier: by a document that should never have been indexed, a bid number the analyser broke apart, a clause cut in half, a filter applied too late, or forty chunks of noise around one useful paragraph. By the time the model sees the evidence, the outcome is mostly fixed.
The way out is to stop thinking of RAG as a component and start treating it as a pipeline: a sequence of stages, each with its own design decision, its own owner and its own way of failing. This part walks the whole pipeline once, so the rest of the series can go deep on the stages that cause the most trouble.
Bergvliet Engineering is fictional, and its systems and figures are illustrative. Its Bid Library assistant now uses hybrid search (Part 2). Relevance has improved. Trust has not yet followed.
Two halves: when content changes, and when someone asks
A retrieval pipeline has two halves that run on different clocks.
The indexing half runs when content changes. Documents are collected, converted to text, split into chunks, enriched with metadata, embedded and written to the index. At Bergvliet this happens when a bid is submitted, a procedure is revised or a contract is signed.
The query half runs on every question. The question is prepared, retrieval runs, results are fused and reranked, checks are applied, context is assembled, an answer is generated, and a record is kept.
Most teams design the query half carefully because that is where the demo lives. Most production failures start in the indexing half, because that is where nobody is watching.
Stage by stage
Ingest. Decide which sources are in scope, which version of each is authoritative, and who owns them. Bergvliet's pilot ingested the whole bid file share, including drafts, superseded revisions and a folder of "old stuff" nobody could explain. The assistant faithfully retrieved all of it. Ingestion is a curation decision before it is a technical one, and No AI without information architecture covers the ownership work it depends on.
Index. Decide the index fields, the language analysers, the chunking rules (Part 4) and the embedding model. Every one of these is hard to change later, because changing it means re-indexing everything. Bergvliet's decision to add an exact-match field for bid numbers and procedure codes came here.
Prepare the query. Establish who is asking, carry their real identity into the request, and extract anything the question contains that should be searched exactly: bid numbers, codes, clause references, names. Some systems also rewrite the question into a clearer search, which helps with vague questions and can hurt with precise ones. Treat rewriting as an option to test, not a default.
Retrieve. Run lexical and semantic retrieval, with the caller's permissions applied as part of the query (Part 5). Decide how many candidates each method returns. Too few and fusion cannot recover a relevant document ranked low; too many and the later stages slow down.
Fuse and rerank. Merge the lists, rerank the short list against the question, and apply a cut-off. Part 6 covers the trade-offs.
Check. Before anything goes to the model, confirm that each surviving chunk is still permitted for this caller now, and that its document is still current. Indexes lag behind sources; this stage catches the gap.
Assemble and answer. Build the context within a budget, include the identifiers the model needs to cite, and instruct the model to say when the evidence does not answer the question. A pipeline with no honest "I could not find that" path will invent one.
Record. Log what was retrieved, what survived each stage, what was shown to the model and what the answer cited. Without this record, nobody can diagnose a bad answer later, and Part 7 depends on it.
Where it actually broke at Bergvliet
When the platform team started logging every stage, they could sort the complaints from the first two months of the pilot by where each fault began. The result surprised them.
Only one column of faults was really about the model. The rest were decided earlier:
- A withdrawn procedure was still indexed, so the assistant confidently described a method Bergvliet no longer used.
- Scanned PDFs from older bids had no text layer. They were "in the index" as empty chunks and could never be found.
- A rate schedule had no recorded owner, so nobody could say whether the figures in it were still valid.
- The analyser split bid numbers into fragments, so a search for one bid matched dozens.
- Afrikaans method statements were never matched by lexical search, because the English analyser was applied to them.
- Fixed-size chunking cut clauses mid-sentence, so a clause's subject and its rule landed in different chunks.
- The model was sent forty chunks when one or two were relevant, and it blended them.
- Permission filtering ran after ranking, which occasionally left a site engineer with nothing usable.
And in the answer column: citations to documents the model had not actually drawn on, fluent answers built on thin evidence, and no instruction to admit when the evidence was missing.
Swapping the model would have fixed almost none of this.
Who owns each stage
A pipeline with eight stages has eight sets of decisions, and they do not all belong to the same people. Bergvliet ended up with this split:
| Stage | Owner | Why them |
|---|---|---|
| Ingest | Bid office and knowledge lead | Only they know which documents are authoritative |
| Index | Platform team | Analysers, fields and embeddings are engineering choices |
| Prepare query, retrieve, fuse, rerank | Platform team, with bid office input | Tuned against an evaluation set the bid office helped write |
| Check | Platform team, rules from document owners | Permission and validity rules belong to the content owners |
| Assemble and answer | Platform team | Prompt, budget and refusal behaviour |
| Record | Platform team, retention set by compliance | Evidence must be kept long enough, and no longer |
The important line is the first. The platform team can build a perfect pipeline over the wrong documents. Content ownership is not an IT task, and RAG makes that visible faster than anything else.
Design for the second question, not the first
Demos test the first question. Production lives on the second: the follow-up, the clarification, the "what about the 2023 version?". Each follow-up runs the query half again, and each run must carry the same identity, apply the same filters and produce the same kind of record. Any path that shortcuts the pipeline, such as a cached answer, a "search again" tool an agent can call, or a summary built from an earlier turn, has to respect the same rules.
This is where retrieval meets agent design. When an agent can call search as a tool, it may call it several times with questions the user never typed. The pipeline does not get to relax because the caller is software. From RAG to agents covers when that step is worth taking at all.
Key points
RAG is a pipeline, not a component. Eight stages, each with its own decision and failure mode.
The indexing half fails quietly. Drafts, withdrawn documents, scanned files and bad analysers are invisible until someone asks the wrong question.
Most bad answers are decided before generation. Log every stage and you will see where.
Every stage needs an owner. Content ownership belongs to the business, not the platform team.
Every path follows the same rules. Follow-ups, caches and agent tool calls must apply the same identity, filters and records.
Next: Part 4, Chunking is a business decision. All parts are on the series page.
How CloudNala can help
We map existing RAG systems stage by stage, from ingestion to the evidence record, and identify where answers are being decided before the model ever sees them. The output is a pipeline design with a named owner and a clear decision for every stage, and a short list of the faults costing you the most trust.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za