Ask a team how they know their RAG system is working and the answer is often a spreadsheet: fifty questions, the assistant's answers, and a column where someone marked each one good, partly good or bad. It is better than nothing. It is also the reason so many teams cannot improve their systems once the easy wins are gone.
A "bad" answer can mean the right document was never indexed, was indexed but not retrieved, was retrieved but ranked too low to reach the model, reached the model but was ignored, or was used correctly but cited wrongly. Those are five different faults with five different owners and fixes. One score hides all of them, and the default fix for an unexplained bad score is to change the model or the prompt, which addresses at most one.
The fix is to measure retrieval separately from the answer, and to do it first. If the evidence was not there, nothing downstream matters. If it was, you know to look at generation. This part shows how.
Bergvliet Engineering is fictional, and its systems and figures are illustrative. By this point its Bid Library uses hybrid search, structure-aware chunks, pre-filtered permissions and a reranker. The team needed a way to tell whether each change actually helped.
Build the evaluation set from real questions
Everything here depends on a set of questions with known good evidence. Bergvliet built theirs over two weeks with the bid office, and it is the most valuable artefact in the project.
180 real questions, collected from pilot logs and from bid managers and engineers, not invented by the platform team. The mix deliberately mirrors real use: about 60 that contain an identifier such as a bid number or clause, about 70 that describe an idea in the user's own words, and about 30 that mix both. Some are in Afrikaans or refer to Afrikaans documents.
Expected evidence for each. For every question, a bid manager or engineer recorded which documents, and where practical which clauses or sections, should come back. Some questions have one right document; some have three. A handful have none, because the honest answer is "we have never done that", and those test whether the assistant admits it.
20 leakage probes, described in Part 5: questions written to match restricted content as strongly as possible, run under paired identities.
The set is versioned and owned by the knowledge management lead. When the corpus or the user population changes, it gets new questions. It never gets questions removed because the system fails them.
Two scorecards, not one
With expected evidence recorded, retrieval can be scored without looking at the generated answer at all.
Retrieval scorecard. These need only the question, what retrieval returned and the expected evidence, so they run automatically in seconds:
- Recall at k: of the evidence that should have come back, how much appeared in the top k results? With k set to the number of chunks the model actually receives (eight at Bergvliet), this is the single most important number, because evidence below the cut-off may as well not exist.
- Precision at k: of the top k results, what share was relevant? Low precision means the model is reading noise, which costs money (Part 6) and invites blended answers.
- Rank of the first relevant result, usually reported as mean reciprocal rank (MRR): for each question take 1 divided by the position of the first relevant result, then average. An MRR near 1 means the best evidence is usually first.
- nDCG, for teams that grade relevance rather than marking it yes or no: it rewards putting the most relevant evidence highest and discounts results further down the list.
- Leakage: the paired-identity probes. This is a pass or fail gate, not an average.
Answer scorecard. These need judgement, so they run on a sample and take longer:
- Groundedness: is every claim in the answer supported by the evidence the model received?
- Citation accuracy: does each cited document actually say what the answer attributes to it? An answer can be grounded and still cite the wrong source, which matters when someone acts on the citation.
- Completeness: does the answer address every part of the question?
- Honest refusal: on questions with no answer in the corpus, does the assistant say so?
Who scores the answers
Retrieval scoring is mechanical. Answer scoring is not, and there are two options, best used together.
People. A bid manager reading 30 answers a fortnight against the evidence is slow, and it is the gold standard. People notice when an answer is technically supported and practically misleading.
A model as grader. A separate model, given the question, the evidence and the answer, can be asked whether each claim is supported and each citation correct. It is fast and cheap enough to run on every change. It is also fallible in ways that are easy to miss: graders can be lenient with fluent answers and inconsistent across runs. Bergvliet calibrated its grader by comparing its verdicts with the bid managers' on the same 60 answers, and only trusted it on the checks where the two agreed closely. The people review continues, at a lower volume, so drift in the grader is caught. A trace is not an evaluation covers the same discipline for agents.
Diagnose from the top down
The point of separating the scorecards is that a wrong answer can be traced to its stage. When a bid manager reports a bad answer, the team asks three questions in order, and the logs from Part 3 answer each one.
Over the first quarter after launch, Bergvliet's illustrative breakdown of reported bad answers was roughly: a quarter were content problems (the evidence was not in the index, or was out of date), a third were retrieval problems, a fifth were ranking or budget problems, and the remainder, a little over a fifth, were generation problems. Only that last group would have been helped by changing the model or the prompt.
What the numbers showed
Running the same evaluation set against each version of the pipeline turned arguments into evidence. Illustratively, recall at 8 on identifier questions went from 38% with the vector-only pilot to 91% after hybrid search, the exact-match identifier field and reranking. On conceptual questions the pilot had already done reasonably well, and the gain was smaller. Precision at 8 rose sharply when the forty-chunk habit was replaced by a reranked cut-off. Groundedness improved mostly as a side effect of precision: with less noise in the context, the model blended fewer sources.
Two changes that looked good in demos were rolled back on the evidence. Query rewriting helped vague questions and hurt identifier questions, so it now runs only when the question contains no identifier. A larger chunk size improved conceptual recall and reduced citation accuracy, because the model could no longer point to a specific clause.
Make it a release gate
An evaluation set that runs only when someone remembers is a report, not a control. At Bergvliet, any change to the index configuration, chunking rules, retrieval settings, reranker, prompt or model runs the retrieval scorecard and the leakage probes automatically. A release is blocked if leakage fails at all, or if recall at 8 drops by more than an agreed margin on any question type. The answer scorecard runs on a sample for every release and in full monthly.
The thresholds are a business decision, made with the bid office: how much worse may the assistant get on conceptual questions in exchange for a large cost saving? Writing that trade-off down before it arises is what keeps it from being made by whoever is in a hurry. Evaluating AI agents and evals that gate real work cover the same idea for systems that take actions.
Key points
One score hides five faults. Index, retrieval, ranking, generation and citation fail differently.
Score retrieval first, and automatically. Recall at k, precision at k and rank of first hit need no judgement and run in seconds.
Leakage is a gate, not an average. Any restricted result returned to the wrong identity fails the release.
Calibrate any model grader against people. Trust it only where it agrees with human reviewers.
Diagnose from the top down. Was the evidence indexed, retrieved, kept? Only then look at the model.
Next: Part 8, Operating RAG after launch. All parts are on the series page.
How CloudNala can help
We help teams build an evaluation set from real questions and expected evidence, wire retrieval and leakage scoring into their release process, and calibrate any model-based answer grading against human reviewers. The result is a knowledge assistant whose improvements, and regressions, show up as numbers before they show up as complaints.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za