When a RAG system misses an answer, the instinct is to give the model more. Retrieve twenty chunks instead of ten. Then forty. Context windows are large now, so why not?
Because every extra chunk has a price, and not only in money. It adds tokens to every call, which costs money and time. It adds noise that the model has to read past, which can make answers worse. And it hides the real problem: if the right evidence was ranked thirtieth, the fix is better ranking, not a bigger bucket.
This part is about the stretch of the pipeline between retrieval and the model: how many candidates to fetch, how to choose among them, how much to send, and how to set those numbers from evidence rather than habit.
Bergvliet Engineering is fictional, and its systems and figures are illustrative. Its Bid Library pilot sent the 40 nearest chunks to the model on every question. By the time the team added hybrid search (Part 2), the forty-chunk habit had become the most expensive part of the system.
Recall is cheap, attention is not
It helps to separate two jobs that are easy to blur.
Recall is making sure the right evidence is somewhere in the candidate list. Retrieval is fast and cheap, so it can afford to be generous here. Fetching a few hundred candidates from an index costs milliseconds.
Precision is making sure that what reaches the model is mostly the right evidence. The model's attention is the expensive, limited resource. Everything you send it competes for that attention, and long contexts with many loosely related passages are where models are most likely to blend sources, miss a detail in the middle, or answer from the wrong document.
The mistake in Bergvliet's pilot was using one number, forty, for both jobs. Forty was too few for recall, because a relevant chunk ranked forty-first by vector search was simply gone, and far too many for precision, because the model was asked to read forty passages to use two.
A funnel, not a bucket
The fix is to make the pipeline a funnel: wide where it is cheap, narrow where it is expensive, with a stage in between that applies judgement.
Bergvliet's tuned pipeline works like this:
- Retrieve wide. Lexical and semantic retrieval each return a generous candidate list, with permissions applied as part of the query (Part 5).
- Fuse to a short list. Reciprocal rank fusion merges the lists, and the top 50 go forward.
- Rerank the short list. A reranker reads each of the 50 against the question and scores how well it answers it.
- Cut off. The top eight are kept, and anything below a minimum reranker score is dropped even if fewer than eight remain.
- Assemble. The surviving chunks, with their identifiers and metadata, go to the model, typically around 4,000 tokens of evidence.
The numbers are Bergvliet's, set from its own evaluation results. Yours will differ. The shape is what transfers.
What the reranker costs, and what it buys
A reranker is slower per document than retrieval, because it reads the question and each candidate together instead of comparing pre-computed representations. That is exactly why it is better at judging relevance, and exactly why you only run it on a short list.
The costs are real and worth naming:
- Latency. Reranking 50 candidates adds noticeable time to each question. For an internal knowledge assistant, a second or so is usually acceptable. For a customer-facing chat, measure it.
- Price. Managed rerankers are usually billed per query or per document scored. Azure AI Search's semantic ranker, for example, is a separately billed premium feature with a free monthly allowance, and it reranks at most the top 50 results (checked on 14 October 2026). Self-hosted cross-encoders cost compute instead.
- Input limits. Rerankers read a limited amount of each candidate. Azure's semantic ranker summarises each document into a bounded input, and anything past the limit is ignored. A long chunk may be judged on its first part only, which is another reason chunking (Part 4) matters.
What it buys is usually worth it: the right evidence near the top, which means fewer chunks need to be sent to the model, which means lower cost and better answers downstream.
The context budget
Every model call has a budget: the instructions, the question, the evidence and room for the answer. Writing that budget down, per question, makes the trade-offs visible.
In Bergvliet's illustrative pilot, evidence was over 90% of every call. Forty chunks of about 500 tokens is around 20,000 tokens of evidence per question, most of it irrelevant. After reranking, eight chunks carry roughly 4,000 tokens, and the whole call is about a quarter of its former size.
That ratio carries straight through to cost, because input tokens are usually the bulk of a RAG system's model bill. At the Bid Library's illustrative volume of around 2,500 questions a week, the difference between 21,500 and 5,500 input tokens per question is the difference between a cost line someone notices and one nobody needs to. It also cut median response time, because the model had less to read.
Two budget rules made the biggest difference:
Set a ceiling, not a target. Eight chunks is a maximum. If only three clear the reranker's minimum score, three go to the model, and if none do, the assistant says it could not find the answer. Padding the context with weak chunks to reach a number is how fluent, wrong answers are made.
Spend tokens on metadata, not repetition. A short header on each chunk, with the bid number, document title, clause or step reference and date, costs a few tokens and lets the model cite precisely. Long overlaps between neighbouring chunks cost many tokens and mostly repeat themselves.
Prompt caching and other levers
Some costs in a RAG call do not change between questions. The instructions and any fixed reference material are the same every time, and several model providers offer prompt caching that charges less for a repeated prefix. Put the stable parts of the prompt first and the variable evidence after, and caching can reduce the cost of the fixed part. It does nothing for the evidence itself, which changes with every question. Measure the cache hit rate rather than assuming it; cache hit rate can hide AI cost explains why a high rate can still sit on top of a growing bill.
Other levers are worth testing on your own evaluation set rather than adopting on reputation: a smaller, cheaper model for straightforward lookup questions with a larger one for synthesis; tighter retrieval for questions that contain an identifier, since an exact bid number rarely needs fifty candidates; and returning the evidence itself, with no generated answer, when the user really wants the document.
Tune from evidence, not from demos
Every number in this part, the candidate count, the short-list size, the cut-off and the minimum score, is a dial. Turned by feel, they drift towards "more", because more always seems safer in a demo.
Bergvliet tunes them against the evaluation set that Part 7 describes. For each setting, the team measures whether the needed evidence still reaches the model, how much irrelevant text comes with it, and what each question costs in tokens and seconds. The chosen settings are the cheapest ones that keep retrieval quality at the agreed level. When the corpus or the question mix changes, they measure again. AI observability and cost covers how to keep that cost visible per case rather than as a monthly surprise.
Key points
More context is not more accuracy. Extra chunks cost tokens, time and the model's attention.
Retrieve wide, send narrow. Recall is cheap at retrieval; precision matters at the model.
Rerank a short list. It costs latency and money per document and usually pays for itself downstream.
Budget every call. A ceiling on evidence, a minimum relevance score and an honest "not found" beat a fixed chunk count.
Set the dials from evidence. Tune candidate counts and cut-offs against an evaluation set, and recheck when the corpus changes.
Next: Part 7, Evaluate what was retrieved before what was written. All parts are on the series page.
How CloudNala can help
We help teams find the cheapest retrieval settings that still deliver the evidence their users need. That means measuring candidate counts, reranking, cut-offs and context size against real questions, and then putting the cost per question somewhere the budget owner can see it. Most of the saving in RAG comes from sending the model less, and sending it the right things.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za