There is a question that comes up in every AI conversation once the initial enthusiasm settles: "But how does it know about our stuff?"
It is the right question. A language model is trained on a large volume of public text. It has never seen your tender library, your standard operating procedures, your policy manual or your client history. Ask it something specific about your organisation and it will do what it always does — produce a fluent, plausible answer — with no way for it, or you, to tell that it is guessing.
The pattern that fixes this is called retrieval-augmented generation, usually shortened to RAG. Almost every "AI that understands our business" product you have been pitched is a RAG system underneath, and the differences between a good one and a bad one are entirely in the details this article covers.
In plain English
RAG does something very simple. Before the model answers, the system goes and finds the handful of passages from your documents that are most likely to be relevant, and puts them in front of the model along with the question. The model then answers using that material rather than from memory.
That is the whole idea. The model does not learn your documents. It reads the relevant bit at the moment it is asked, the way a consultant would open the right page before answering.
This distinction matters commercially, because it is the source of a persistent misunderstanding. Organisations often assume they need to "train a model on our data", imagine the cost and the risk of that, and stall. In the overwhelming majority of business cases, no training is involved. Your documents stay where they are, under the access controls they already have, and the system reads from them on demand.
The architecture view
There are two separate paths, and confusing them is the source of a lot of muddled thinking.
The top path runs once per document. Text is extracted from the file, split into chunks of a few hundred words, and each chunk is converted into an embedding — a list of numbers that represents its meaning — which is stored in a vector database alongside a pointer back to the original source.
The bottom path runs every time someone asks something. The question is converted into an embedding using the same method, the database returns the chunks whose meaning sits closest to it, and those passages are handed to the model with an instruction along the lines of "answer using only this material, and cite where each claim came from".
The important property of embeddings is that they match on meaning rather than words. A question about "what certificates do we need to submit" will retrieve a clause headed "mandatory supporting documentation", even though the two share almost no vocabulary. That is why RAG feels smarter than the search box in your document management system, and it is the entire technical advance involved.
Why it fails
Nearly every RAG failure traces back to the top path, not the bottom one. The model is rarely the problem. Some patterns to expect:
The PDF was a photograph. A scanned tender document with no text layer extracts as nothing, or as garbage. If half your source library is scanned, you have an optical character recognition project before you have an AI project, and it needs to be scoped and paid for.
The chunks were split badly. Cut a document every five hundred words without regard for structure and you will eventually split a table from its heading, or a requirement from the clause that qualifies it. The retrieved passage then looks relevant and is misleading — the worst combination. Chunking along the document's own structure, and carrying section headings into each chunk, fixes most of this.
The library contains three versions of the truth. The 2023 policy, the 2025 draft revision, and a partially-updated copy someone saved to a shared drive. Retrieval will cheerfully return the wrong one. No amount of model quality compensates for a source library nobody owns.
There are no citations. If the answer does not show which document and which section it came from, nobody can check it, and it will therefore be checked by nobody until it causes a problem. Citations are not a nice-to-have feature; they are the mechanism by which the system becomes auditable.
Everyone can see everything. If retrieval runs across the whole estate with a single service account, an employee asking an innocent question can receive content from a folder they have no right to. Access control has to be applied at retrieval time, filtered by the identity of the person asking — not applied to the answer afterwards.
The question was outside the documents. Asked something the library does not cover, a poorly configured system will answer from general knowledge without saying so. The correct behaviour is to say it does not know. That is a prompt and evaluation decision, and it needs to be tested deliberately.
A practical example
Take a tender pack — a few hundred pages across a main document, annexures, pricing schedules and a bidder's declaration.
Indexed properly, a bid manager can ask "what are the mandatory returnables?" and get a list where each item cites the clause it came from. They can ask "does this tender allow a joint venture?" and get either the relevant clause or an honest "this is not addressed in the documents provided". They can ask "what changed between this tender and the addendum?" if both are in the index.
What they cannot get from RAG alone is any action. The system knows the tender requires a valid tax clearance certificate; it has no idea whether you have one, when it expires, or who to ask. That gap — between knowing and doing — is the subject of article four in this series, and it is where agents come in.
How to start small
Resist the urge to index everything. A RAG system over three hundred well-maintained documents that one team relies on daily will outperform one over forty thousand documents of uncertain provenance, and it will be far easier to evaluate.
Pick a document set with a clear owner. Confirm the files have real text in them. Index it, put citations on every answer, and give it to ten people who know the material well enough to catch it being wrong. Write down the twenty questions they ask most, along with the correct answers, and use that as your test set from then on. That test set is worth more than any model upgrade you will make in the first year.
Then measure something honest: how long it used to take to find these answers, and how often the system now gets them right on the first attempt. If you cannot show both numbers, you have a demo rather than a deployment. Building the smallest backend that does this properly is covered in the next article.
What this really depends on
RAG is a retrieval pattern, not a form of intelligence. Its output quality is bounded almost entirely by things that have nothing to do with AI: whether your documents are current, whether they are machine-readable, whether anyone owns them, and whether the permissions on them reflect reality.
Which is an uncomfortable finding for a lot of organisations, because it means the AI project surfaces a document governance problem that predates it by a decade. That is not a reason to avoid the project. It is a reason to scope it with the document work included, rather than discovering it in month three.
How CloudNala can help
We help organisations get the unglamorous half of this right — assessing whether a document set is actually usable, sorting out extraction and chunking for the formats you really have, wiring retrieval to your existing permissions rather than around them, and building the evaluation set that lets you prove the answers are correct. The AI part is usually the fastest piece; the data foundations are where the project is won or lost, which is a theme we return to in our work on data readiness.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za