Most retrieval-augmented generation projects start the same way. Someone loads a folder of documents into a vector database, connects a large language model, and asks a few questions in a meeting. The answers are fluent, the sources look relevant, and the room agrees to a pilot.
Then the pilot meets real users. An engineer types a project number and gets back four documents about similar projects and none about the one they named. A lawyer asks about clause 14.3 and receives a sensible paragraph about liability from a different contract. Nobody can say exactly what went wrong, so the first instinct is to try a bigger model.
The model is rarely the problem. A good model cannot repair missing, stale or unauthorised evidence, and it cannot answer from a document that retrieval never handed it. This eight-part series is for teams that have moved past "upload a PDF and ask a question" and now need retrieval that is relevant, permission-aware, measurable and operable. It starts with the most common early mistake: treating vector search as the whole of retrieval.
If you are new to the pattern itself, RAG explained simply and the minimum useful RAG backend cover the basics. This series assumes those and goes further.
The engineering firm that searched by meaning
Bergvliet Engineering is fictional, and its systems and figures are illustrative. Bergvliet is a consulting engineering firm in Centurion with about 350 staff, working on civil, structural and water projects for municipalities, provincial departments, state-owned entities and private developers. Over 18 years it has accumulated roughly 60,000 documents that matter to bidding: tender submissions, method statements, standard procedures, contract clauses, rate schedules, lessons-learnt notes and meeting minutes. Some of the older method statements are in Afrikaans.
Every bid is numbered internally in the form BE-T-2024-0219, and every client tender has its own reference as well. Standard procedures carry codes such as SP-031. Engineers quote specifications by code, such as SANS 1200 D for earthworks.
The bid office, six bid managers and a knowledge management lead, wanted an assistant that could answer questions such as "what did we commit to on dewatering in BE-T-2024-0219?" or "have we priced night work on a road rehabilitation job before?" A four-person platform team built the first version of the Bid Library in six weeks. It split every document into 1,000-character chunks, turned each chunk into an embedding, stored the embeddings in a vector index, and sent the 40 nearest chunks to a language model with the question.
In the demo it was impressive. Two months into the pilot, the bid managers had stopped using it for anything that mattered.
What vector search actually does
An embedding model turns a piece of text into a long list of numbers that positions it in a "meaning space". Texts about similar things land near each other, even when they share no words. Vector search takes the question, turns it into the same kind of list, and returns the stored chunks that sit closest to it.
That is a genuinely useful capability. Ask about "keeping trenches dry" and vector search can find a method statement that only ever says "dewatering". Ask in English and a multilingual embedding model may find a relevant Afrikaans procedure. Paraphrase, synonyms, loose descriptions of a problem: this is where semantic search earns its place.
But notice what "close in meaning" leaves out. A bid number has almost no meaning. BE-T-2024-0219 and BE-T-2023-0107 look nearly identical to an embedding model, and both sit near every other document about the same kind of work. The model is doing exactly what it was built to do. It is just answering a different question from the one the engineer asked.
Nearby is not the same as right
The Bid Library's failures followed a pattern the platform team only saw once it started logging what retrieval returned.
When the question named a specific bid, the vector index returned a tidy list of dewatering method statements, all relevant to the topic and all from the wrong projects. The method statement for BE-T-2024-0219 was in the index. It ranked ninth. The pilot did send it to the model, as one of forty passages, but eight passages about other bids' dewatering commitments came first, and the model answered from the majority. The answer described what Bergvliet usually commits to on dewatering, which is not what it committed to on that bid. Read quickly, it looked right.
The same pattern appeared everywhere precision mattered:
- Codes and identifiers. Specification codes, clause numbers, procedure numbers, tender references and drawing numbers carry their meaning in the exact characters.
- Names. People, clients, towns and project names blur together in meaning space. "Who was the resident engineer on the Bronkhorstspruit pipeline?" needs the exact name of the project.
- Rare terms. A specialist term that appears in three documents out of 60,000 is precisely what the user wants found, and precisely what a dense model may not represent distinctly.
- Negation and exceptions. "Shall not be liable unless" and "shall be liable" are close in meaning space. In a contract they are opposites.
None of these are exotic. In a professional services firm, a bank, a municipality or a law practice, a large share of real questions contain at least one of them.
The opposite tool has the opposite blind spot
The obvious alternative is the search engine organisations have used for decades: lexical, or keyword, search. Most modern lexical search ranks results with BM25, a scoring function from the probabilistic retrieval research of Stephen Robertson, Karen Spärck Jones and colleagues. In plain terms, BM25 rewards a document for containing the query's words, rewards rare words more than common ones, and stops rewarding repetition after a point so that a document cannot win simply by saying "dewatering" forty times.
Lexical search finds BE-T-2024-0219 instantly, because the token is there. It finds SANS 1200 D, clause 14.3 and the Bronkhorstspruit pipeline. And it fails exactly where semantic search succeeds: ask about "keeping trenches dry" and it returns nothing about dewatering, because the words differ. Ask in English about an Afrikaans procedure and it finds nothing at all.
So the choice is not between an old tool and a new one. They are two instruments with complementary blind spots, and a business corpus needs both. Part 2 explains how to combine them without one drowning the other.
Vector search is not a knowledge strategy
There is a broader lesson in Bergvliet's pilot. The team had chosen a retrieval technology and assumed it amounted to a retrieval design. The questions that decide whether a knowledge assistant can be trusted were never asked:
- Which kinds of question will people actually ask, and what proportion contain identifiers?
- Which documents are authoritative, which are drafts, and which have been withdrawn?
- Who may see what, and does the index know?
- How will we know whether the right evidence came back, separately from whether the answer sounded right?
- Who owns the content once the pilot team moves on?
These questions are what the rest of this series is about. A vector database answers none of them. Neither does a bigger model.
A quick test for your own pilot
Before redesigning anything, collect 30 real questions from the people who will use the assistant, not from the team that built it. Then sort them:
| Question type | Example at Bergvliet | Likely to need |
|---|---|---|
| Contains an identifier or code | "Dewatering commitments in BE-T-2024-0219" | Lexical |
| Describes an idea in the user's own words | "How do we usually handle wet-season earthworks?" | Semantic |
| Contains a name | "Projects with the Bronkhorstspruit pipeline team" | Lexical |
| Crosses languages | English question, Afrikaans procedure | Semantic, multilingual |
| Mixes both | "Night-work rates on road jobs since BE-T-2022-0388" | Both |
At Bergvliet, an illustrative count put 41% of real questions in the first or third rows. Any pilot that handles those badly will lose the users who matter most, because they are the people asking precise questions about real work.
For each question, write down the document that should come back. That list is the start of the evaluation set Part 7 builds properly, and it is the cheapest diagnostic you will ever run.
Key points
A good model cannot recover a missing document. If retrieval does not hand over the right evidence, generation cannot fix it.
Vector search finds meaning, not identity. Bid numbers, codes, clause numbers and names are where it struggles.
Lexical search has the opposite blind spot. It finds exact tokens and misses paraphrase and cross-language matches.
Business corpora need both. Most real question sets mix identifiers with descriptions of ideas.
Retrieval technology is not retrieval design. Ownership, permissions, freshness and evaluation are decisions, not features of a database.
Next: Part 2, Hybrid search in plain language. All parts are on the series page.
How CloudNala can help
We review knowledge assistants that work in a demo and disappoint in use. We start with a sample of real user questions and the documents that should answer them, measure what retrieval actually returns, and show where the gap is: content, search method, ranking, permissions or the answer itself. That gives you a short, evidence-based list of fixes rather than a model upgrade that changes nothing.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za