Lindiwe Mokoena runs the programme management office at the Blouberg Infrastructure Programme, a fictional multi-year public programme that builds and refurbishes roads, bridges and stormwater works across about forty sites. Its people, systems and figures are illustrative. Her team had piloted an assistant over the programme's documents, and she tested it with the question her regional manager asks most often: "Has the contractor on Site 14 been paid for milestone 3, and was the work verified?"
The assistant answered in two confident sentences. Yes, the milestone had been paid, and yes, an inspection had verified it. It cited three sources. The first was a monthly progress report that said a payment certificate for milestone 3 had been submitted, which is not the same as paid. The second was an inspection report for milestone 3 on Site 41, a different bridge with almost identical wording. The third was the original contract schedule, which a variation order had amended four months earlier, moving part of milestone 3's scope into milestone 4. Every passage was relevant to the question. None of them answered it.
Nothing in the pilot was broken in the usual sense. The search found text close to the question, as designed. But the question was not really about text. It was about a particular contractor, a particular site, a particular milestone as it stood after a change, and a payment status that lives in a finance system, not in a report. This series is about the architecture that answers questions like that. Retrieval is one pattern inside it.
What the question actually contained
Read the regional manager's question slowly and it breaks into parts that have nothing to do with wording.
- Identity. "The contractor on Site 14" is a joint venture called one thing in the contract, another on its invoices and a third as a supplier number in the finance system. "Site 14" is a project code in one system and a polygon identifier in the geographic information system.
- Relationships. The contractor holds a contract; the contract covers the site; milestone 3 belongs to that contract; a payment certificate certifies a milestone; an inspection verifies one.
- Vocabulary. "Paid" could mean certified by the engineer, approved for payment or money that has left the account. Each is a different event, recorded in a different place.
- Effective time. Milestone 3 means the milestone as amended by the variation order, not as first written.
- Authority. The finance system is authoritative for payment status. A progress report that mentions a payment is a secondary account of it.
- Permission. Payment amounts are visible to the finance and contracts teams, not to every site engineer who might ask.
- Provenance. The answer has to show which record supports each part, and as at which date.
A retrieval pipeline that returns the five most similar passages does none of this by itself. Why vector search alone is not enough explains why nearby wording is not the same as the right evidence. The point here is wider: even a perfect retrieval of the right passage is not enough when the passage is not the authoritative statement of the fact.
Similarity is a property of text; trust is a property of sources
Similarity search measures how close two pieces of text are. It knows nothing about which system owns a fact, whether a statement has been superseded, or whether two names refer to the same company. Those are properties of the organisation's information, and they have to be modelled somewhere outside the embedding.
The useful shift for Lindiwe's team was to stop asking "how do we retrieve better?" and start asking "what does a trusted answer to this kind of question require?" For each class of question, they wrote down an answer contract:
- which identifiers in the question must resolve to exactly one record before anything else happens;
- which system of record is authoritative for each fact the answer will state;
- which date the answer is valid for, and which version of a changing document applies on that date;
- which parts may be shown to which roles; and
- what evidence must be attached to each statement.
Once those contracts existed, it became obvious that most of the regional manager's question should never have gone to similarity search at all. Payment status is a lookup. Milestone scope is a lookup against the contract as amended. Only the inspection narrative, what the inspector actually observed, needs retrieval over text, and even then the search should be constrained to the right site and milestone.
Seven things a trusted answer needs
The answer contracts at Blouberg kept returning to the same seven needs. They make a practical checklist for any enterprise question.
| Need | What goes wrong without it | Where it lived at Blouberg |
|---|---|---|
| Identity | Two contractors, or two sites, are merged or confused | A new identifier register linking contract, finance and site codes |
| Relationships | The answer cannot connect a payment to a milestone | Contract and finance records, joined on governed keys |
| Vocabulary | "Paid", "complete" and "verified" mean different things to different teams | A short glossary owned by the contracts manager |
| Effective time | Superseded scope or rates are presented as current | Variation orders, with the date each takes effect |
| Authority | A report's description of a fact outranks the system that holds it | A source register naming the authority for each fact type |
| Permission | Restricted figures reach the wrong role | The source systems' own access rules, carried into every lookup |
| Provenance | Nobody can say why the answer said what it said | Evidence identifiers and as-at dates attached to every statement |
None of these is new to information management. No AI without information architecture makes the case that this layer decides whether answers can be trusted, and The semantic layer is your hallucination control shows how definitions protect numerical questions. What changes with an assistant is that every gap becomes visible at once, in a fluent sentence, to someone who will act on it.
The architecture around retrieval
If retrieval is one pattern inside a wider architecture, it helps to see the whole thing. The layers below are the progression this series follows.
The lower layers are about what goes in: which sources are governed and by whom, how text, tables and images are extracted, and which metadata and permissions travel with each item. RAG That Survives Production covers these for documents in depth, from the retrieval pipeline end to end to permission-aware retrieval. This series does not repeat that work.
The middle layers are the ones most pilots skip: a register of identities, a semantic model of the organisation's vocabulary, and a store of relationships, which may be relational tables, a graph or both. These are what turn "similar text" into "the right thing". The upper layers are about what comes out: a query plan that sends each part of a question to the right place, generation that is tied to evidence, and evaluation, lineage and correction so the system stays trustworthy as content changes.
The rest of the series takes the middle and upper layers in turn: when a knowledge graph is justified, how to write an ontology a delivery team can use, how to trace an answer back to its sources and how to correct an error without leaving copies behind. Part 3 returns to the lower layers for what changes with tables, photographs, drawings and maps.
What Blouberg changed
The team did not start by tuning chunk sizes or swapping the embedding model. They made four changes, each small.
An identifier register. For the three entity types that appeared in most questions (contractors, sites and milestones), they built a table that links each real-world thing to its codes in every system. A contractor has one programme identifier, and the register records its contract name, its invoice names and its supplier number. Building it took two people about three weeks, most of it spent resolving joint ventures and renamed sites.
A question planner. Before any retrieval, the assistant extracts the identifiers in a question and resolves them against the register. If "Site 14" resolves to one site, the planner continues. If it resolves to none or several, the assistant asks which one was meant instead of guessing.
Facts from systems, narrative from documents. Payment status, certificate dates and amended milestone scope now come from lookups against the contract and finance systems, with the caller's permissions applied. Retrieval is used for what documents are good at: the inspector's observations, the reasons given in a variation order, the wording of a policy. Retrieval queries are filtered by the resolved site and milestone identifiers, so a near-identical report from Site 41 can no longer appear.
An answer that shows its parts. The reply to the regional manager's question now has three short lines: payment status from the finance system as at a stated date, verification status with the inspection report it relies on, and a note that milestone 3 was amended by a named variation order. Each line carries its source.
On the team's internal set of forty real questions, which mixes lookups, narrative questions and a few deliberately ambiguous ones, correct answers with the right evidence rose from 14 of 40 to 31 of 40 (illustrative figures for a fictional organisation). The more important result was qualitative: wrong answers became refusals or clarifying questions instead of confident sentences.
When basic RAG is enough
None of this means every assistant needs an identifier register and a semantic model. Plenty of useful questions are about one document: what the supply chain policy says about emergency procurement, what the standard specification requires for culvert backfill, how to submit a site diary. For those, good retrieval with sound metadata is the right design, and adding more machinery would only add cost.
A quick way to tell the difference is to look at the question, not the corpus.
| If the question... | Basic retrieval may be enough | You need more of the architecture |
|---|---|---|
| Names a specific contractor, site, person or asset | Rarely | Usually: identity must resolve first |
| Asks for a status, amount or date | No | Yes: look it up in the system of record |
| Uses a term teams define differently | No | Yes: the vocabulary must be agreed |
| Is about one current document | Yes | Only if versions or permissions are complex |
| Connects several records ("which", "all", "affected by") | No | Yes: relationships must be modelled |
| Must stand up to an auditor | Rarely | Yes: provenance and as-at dates are required |
For a public programme, the last row matters more than it looks. Public spending is audited, and an answer that cannot show its source and its date is a liability when it is later quoted in a memo or a meeting.
A first-month checklist
- Collect thirty real questions from the people who will use the assistant, not from the project team.
- For each question, mark which of the seven needs it depends on. The pattern will show which layers matter.
- Write a source register: for each fact type, the authoritative system and its owner.
- Pick the two or three entity types that appear most often and build an identifier register for them.
- Write a glossary entry for every term in the questions that two teams use differently.
- Change the assistant so it resolves identifiers before retrieving, and asks when it cannot.
- Make every answer show its sources and as-at date, then test with the same thirty questions, scoring retrieval and the answer separately as described in Evaluate what was retrieved before what was written.
Key points
- Enterprise questions usually depend on identity, relationships, vocabulary, effective time, authority, permission and provenance, not only on wording.
- Similarity is a property of text; trust is a property of sources. Model the second outside the embedding.
- Write an answer contract for each class of question: which identifiers must resolve, which system is authoritative and which date applies.
- Send facts to systems of record and use retrieval for narrative evidence, filtered by resolved identifiers.
- Basic retrieval is enough for single-document questions. Add layers only when the questions require them.
Next: Part 2, When you need a knowledge graph. All parts are on the series page.
How CloudNala can help
CloudNala's knowledge architecture review starts from the questions your people actually ask. We classify a sample of them against the seven needs in this article, map each fact type to its authoritative source and owner, identify the entity types that need an identifier register, and set out which questions should go to lookups, which to retrieval and which need more structure, so that you invest in the layers your questions require and no others.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Knowledge Architecture Review or write to us at consult@cloudnala.co.za