Once an organisation accepts that connecting AI to its own documents is a retrieval problem rather than a training problem, the next question is architectural: what do we actually have to build?
The honest answer is less than most vendors imply and more than most first attempts include. This article describes the smallest version that is genuinely useful in production, and — just as importantly — names the sophisticated components you should deliberately leave out of version one.
The parts you cannot skip
Two endpoints. Four moving parts. One log.
An upload path. Something has to accept a document, pull the text out of it, split it sensibly, turn the chunks into embeddings, and store them. In practice this is where the engineering effort concentrates, because real document estates contain scanned PDFs, spreadsheets pretending to be reports, and files whose structure only makes sense to the person who wrote them.
A store. A vector database holds the embeddings and returns nearest matches. This can be a dedicated service or, very commonly and very sensibly, a vector extension on the relational database you already run. If your organisation already operates PostgreSQL, adding vector search to it is a smaller decision than adopting a new piece of infrastructure, and it keeps your document metadata and your embeddings in the same place.
An answer path. Take a question, retrieve the closest chunks, assemble them into a prompt with clear instructions, call the model, return the answer together with the sources it used.
A log. Every question, the chunks that were retrieved, the answer given, the latency and the cost. This is not observability sophistication — it is a table. But without it you cannot debug a single complaint, and you cannot tell whether last week's prompt change helped or hurt.
The shape of a request
Written as a flow, the answer path is short enough to hold in your head:
flowchart TD
Q[Question from a named user] --> P[Apply that user's permissions]
P --> R[Retrieve top matching chunks]
R --> C{Anything relevant found?}
C -- No --> N[Say so — do not answer from general knowledge]
C -- Yes --> A[Model composes answer from those chunks only]
A --> S[Attach citations]
S --> L[Log question, sources, answer, cost]
L --> O[Return to user]The decision diamond is the part first attempts leave out, and it is the one that determines whether people trust the system. A retrieval that finds nothing relevant should produce "I could not find this in the documents available to you", not a fluent paragraph assembled from the model's general knowledge. Getting that behaviour reliably takes an explicit instruction, a relevance threshold, and a test case that checks for it.
Note also where permissions sit: before retrieval, not after. Filtering the answer after the fact means the sensitive content was already retrieved and put in front of the model, which is exactly what you were trying to prevent.
What to leave out of version one
This is the more useful half of the advice, because the failure mode here is not building too little. It is building an architecture you cannot evaluate.
Leave out reranking models, hybrid search tuning, query rewriting, multi-agent routing, knowledge graphs, fine-tuning and semantic caching. Every one of them is a real technique that solves a real problem. None of them solves a problem you have yet identified, and each one adds a variable that makes it harder to work out why an answer was wrong.
The sequence that works is: build the simple version, create a test set of questions with known correct answers, find out where it actually fails, then add the specific component that addresses that specific failure. Teams that skip to the sophisticated architecture end up with a system that is wrong in ways they cannot diagnose, because they have five interacting components and no baseline.
Choices that matter more than they look
A few decisions in this small system have consequences far beyond version one.
Chunk with the document's structure, not against it. Splitting on headings and keeping the heading trail inside each chunk costs a day of work and improves retrieval quality more than most model upgrades. It also makes citations meaningful — "section 4.2, Mandatory Returnables" rather than "page 37".
Store the source pointer at index time. Every chunk should carry the document identifier, version, section and access classification. Retrofitting this later means reindexing everything, and you will want it the first time someone asks whether an answer came from the current policy or the superseded one.
Make reindexing routine. Documents change. If updating the index is a manual task someone remembers to do, the system will quietly drift out of date and nobody will notice until it gives a confidently wrong answer from a superseded document. Reindexing should be triggered by the document changing, not by a person.
Keep the model swappable. Put the model call behind a thin interface so that changing provider or version is a configuration change. Model pricing and capability have moved substantially every few months for several years, and there is no reason to expect that to stop.
A South African deployment consideration
Two constraints come up locally often enough to design for from the start.
The first is data residency. If your documents carry personal information subject to POPIA, or contractual restrictions on where they may be processed, that shapes which model providers and which regions are available to you. This is a question to answer in week one, because it constrains the architecture rather than decorating it. Both major hyperscalers now operate South African regions, and several model providers offer regional or in-country processing options — but the combination you need must be confirmed rather than assumed.
The second is connectivity. If the people who will use this are in district offices, on site, or on mobile data, then a system that assumes a fast, stable connection will be judged as broken. Keep responses small, stream them where you can, and make the system usable on a bad link.
How to know it is working
Before you have opinions about the system, get numbers.
Assemble thirty questions your target users genuinely ask, with correct answers agreed by someone who knows the material. Run them. Record how many were answered correctly, how many cited the right source, how many were correctly refused because the documents did not cover them, and how many were confidently wrong. That last number is the one that matters — a system that is occasionally unhelpful is annoying, while a system that is occasionally confidently wrong is dangerous.
Re-run that set before every change. It takes minutes once it exists, and it converts "the new prompt feels better" into something you can put in front of a risk committee. We go further into this in the article on evaluation.
How CloudNala can help
We build these backends the boring way — extraction that copes with the documents you actually have, retrieval wired to your existing identity and permissions, citations on every answer, a log you can audit, and a test set that tells you whether it works before your users do. Usually deployed into your own cloud environment, so the documents and the traces stay inside your governance boundary.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za