A familiar conversation happens in AI planning meetings. Someone says the assistant needs to "know" the organisation's information. Someone else says retrieval is the standard way to do that. A third person has read that fine-tuning produces better results, and a fourth has seen a demonstration of a knowledge graph and wants one. The discussion becomes a choice between techniques, and the choice is made on enthusiasm or vendor positioning.
The better starting point is the question the system must answer. Retrieval, fine-tuning and knowledge graphs are not three ways to do the same job. They do different jobs. Choosing well means identifying which job you actually have, and accepting that many questions need none of them.
CloudNala's position is direct: route the question before you pick the technique. If the answer depends on exact numbers, query the data. If it depends on facts that change, retrieve them. If it depends on relationships across many hops, model the relationships. If the gap is behaviour or format rather than knowledge, try examples first and fine-tune only if they fall short.
Gariep Grain Co-operative is fictional, and its systems and figures are illustrative. By this point in the series, Gariep had a member assistant drawing on a proper deliveries data product (Part 1) with an agreed definition of "delivered" (Part 2). The next wave of requests from the business was broader, and each request came with a technique already attached.
Four kinds of question
Gariep's operations, finance and member-services teams brought a list of things they wanted AI help with. Grouped by what the answer depends on, the list fell into four kinds.
Questions about exact numbers. "What is my graded tonnage this season?" "How many loads arrived at Wesselsbron yesterday?" "What was the average moisture on maize this week?" The answer is a calculation over structured records. A language model should not compute it from text; it should query the data and report the result.
Questions about facts that change. "What is the current moisture deduction table?" "When does the final pool payment run?" "What documents do I need to register a new farm?" The answer lives in policies, notices and procedures that are updated several times a season. The model needs the current version at the moment it answers, and the member should be able to see where the answer came from.
Questions about relationships. "A mill has complained about off-odour in a rail shipment. Which members delivered into the bin it came from?" The answer requires following a chain of links across systems: shipment to silo bin, bin to loads, loads to members. No single document contains it.
Requests about behaviour or format. "Write the grading note for each sample in our standard format, with the defects in the order the graders use." Here the model already has the facts it needs, the sample results. What it lacks is a consistent way of writing them up.
Matching technique to question
Query the data for exact numbers. This is not an AI technique at all, which is why it is often overlooked. A language model can translate the member's question into a call to a governed query interface or a semantic layer, then phrase the result. The number comes from the database, not from the model. The semantic layer is your hallucination control explains why this matters. At Gariep, most member questions turned out to be this kind.
Retrieval (RAG) for facts that change. The current version of a policy is retrieved at query time and placed in the model's context, so the answer reflects today's document and can cite it. Updating the knowledge means updating the document, not retraining anything. Retrieval is the right default for policies, procedures and notices. The RAG That Survives Production series covers what it takes to make retrieval work reliably, from why vector search alone is not enough to operating RAG after launch.
A knowledge graph for relationships. Entities (members, loads, bins, shipments, buyers) and the relationships between them are stored explicitly, so a question can be answered by walking the links. Graph-based retrieval approaches, such as Microsoft Research's open-source GraphRAG, combine this with language models. Part 4 looks at this in detail, including its considerable costs.
Examples, then fine-tuning, for behaviour and format. Fine-tuning adjusts a model's weights using training examples, so it learns a style, a format or a narrow task. It is good at that. It is poor at keeping facts current, because new facts mean new training, and it cannot show where an answer came from. Before fine-tuning, put a handful of good examples in the prompt. For Gariep's grading notes, ten examples in the prompt produced the right format on almost every sample. Fine-tuning would have added a training pipeline to maintain for little gain.
The common mistakes
Fine-tuning to teach facts. This is the most expensive mistake. A model fine-tuned on last season's policy documents will answer confidently from last season's policies, with no citation, and the only fix is to train again. If the knowledge changes, retrieve it.
Retrieval for numbers. Placing tables or reports into a retrieval index and asking the model to read off figures works in a demonstration and fails in production. The model may pick the wrong row, the document may be stale, and nothing stops it doing arithmetic of its own. If the answer is a number, query the system that holds it.
A knowledge graph because it looks sophisticated. Graphs carry real cost: entity extraction, resolving duplicates, schema design and ongoing maintenance. If your questions are mostly single-hop ("what does policy X say?"), a graph adds little. Use one where multi-hop questions are frequent and valuable.
Treating the choice as exclusive. Real systems combine techniques. Gariep's eventual member assistant queried the deliveries product for numbers, retrieved current policies for procedural questions, and handed traceability questions to an operations tool backed by a graph, not to the member assistant at all.
A worked routing table
The routing exercise at Gariep produced a simple table that the product owner kept as the assistant's scope document:
| Question | Depends on | Route |
|---|---|---|
| My tonnage this season | Deliveries product | Query, member-facing view |
| Current moisture deductions | Policy notice | Retrieval, cite the notice |
| When is the final payment | Payment calendar | Retrieval, cite the calendar |
| Which members are linked to a complaint | Shipment, bin and load records | Graph, operations staff only |
| Draft a grading note | Sample results | Prompt examples, no fine-tune |
| Advice on whether to sell now | Market judgement | Out of scope, refer to a person |
The last row matters as much as the others. Part of routing is deciding which questions the system should not answer.
The trade-off
Every technique you add is something to operate. Retrieval needs a document pipeline, permissions that match the source, and evaluation. A graph needs extraction, resolution and a schema that evolves. Fine-tuning needs training data, a training pipeline and re-evaluation whenever the base model changes. Querying the data needs a governed interface and a semantic layer.
The cheapest architecture is the one with the fewest techniques that still covers your questions. That is usually querying plus retrieval. Add the others when a class of question that matters cannot be answered without them.
The decision
Before choosing a technique, list twenty real questions your users will ask. Sort them by what the answer depends on: exact numbers, changing facts, relationships, or behaviour. Count each pile. The largest pile tells you where to start, and an empty pile tells you what not to build.
Next: Part 4, Knowledge-graph search for relationship-heavy questions. All parts are on the series page.
How CloudNala can help
We help teams sort their AI use case's questions by what the answers depend on, then design the smallest combination of querying, retrieval and, where justified, graphs or fine-tuning that covers them. The result is an architecture that the team can operate, with a clear scope of what the system will and will not answer.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Data Readiness Review or write to us at consult@cloudnala.co.za