Before a document can be searched by a RAG system, it is cut into pieces. The pieces are called chunks, and they are what retrieval actually finds and what the model actually reads. A user never sees a whole document in a RAG answer. They see whatever survived the cut.
In most pilots the cutting rule is a single number: 500 tokens, 1,000 characters, sometimes with a small overlap between neighbouring chunks. The number is chosen once, usually from a tutorial, and never revisited. It treats a liability clause, a rate table, a step-by-step procedure and a page of meeting notes as the same kind of thing.
They are not, and the difference shows up as answers that are technically sourced and practically wrong. This part argues that chunking is a decision about your business content, which is why the people who understand the content should help make it.
Bergvliet Engineering is fictional, and its systems and figures are illustrative. Its Bid Library pilot cut all 60,000 documents into 1,000-character chunks. Part 3 listed what broke. Several of the worst faults began here.
What a chunk has to do
A chunk has two jobs, and they pull in different directions.
It has to be findable: small and focused enough that, when it matches a question, it matches because of what it says rather than because it happens to contain a lot of loosely related text.
It has to be self-sufficient: large and complete enough that, when the model reads it on its own, it carries the meaning the author intended, including the conditions, exceptions and context that make it true.
Fixed-size chunking ignores both jobs. It cuts wherever the counter runs out, which is sometimes between two paragraphs and sometimes in the middle of "shall not be liable unless".
The clause that lost its exception
A Bergvliet bid manager asked the Bid Library whether the firm had previously agreed to delay penalties without any notice requirement. The assistant answered confidently, and wrongly.
The relevant contract clause said, in illustrative terms, that the contractor shall not be liable for delay penalties unless notice was given within a set period. The fixed-size splitter had cut it after "The Contractor shall". Retrieval found the second half, which began mid-sentence with "not be liable for delay penalties unless notice was given": no subject, no clause number, no contract reference. The model had to guess who was not liable and under which contract. It guessed the client, and attributed the clause to a different bid whose chunks happened to rank nearby.
Overlap, the usual defence, did not help. A 100-character overlap shares a sentence fragment between neighbours; it does not guarantee that the subject, the rule, its exception and its clause number end up in the same piece. The fix was to split on the document's own structure, clause by clause, and to attach the parent heading and contract reference to every chunk so that a clause read on its own still says where it comes from.
Different content, different rules
Once the team looked at the corpus by content type rather than as one pile of text, the rules more or less wrote themselves.
Contract clauses split along clause and sub-clause numbers. Each chunk carries its clause number, the contract it belongs to, the contract version and its parent heading. A clause that refers to "the period stated in clause 14.1" keeps that reference visible, and the assistant can retrieve 14.1 as well.
Rate schedules and tables are the worst case for fixed-size chunking, because a table cut in half loses its column headers. A row reading "Night work | 1.35 | m³" means nothing without "Item | Factor | Unit" above it. Bergvliet keeps small tables whole and splits large ones into row groups, repeating the header row in every group. Each chunk also carries the schedule's validity date, because an old rate presented as current is a commercial risk.
Standard procedures split along their numbered steps, in groups that keep a complete action together. Safety warnings attached to a step travel with that step, never in a separate chunk. Each chunk carries the procedure code and revision.
Method statements in bids split by section heading. Each chunk carries the bid number, the client, the year and the bid outcome, which lets the assistant say "this is what we proposed on a bid we lost" rather than presenting it as standard practice.
Meeting notes are mostly noise around a few decisions. Bergvliet chunks them around decisions and action items, with date, attendees and project attached, and leaves the discussion text out of the index altogether.
Metadata does half the work
The right column of that diagram matters as much as the middle one. A chunk is only self-sufficient if it knows where it came from. Attaching metadata gives four benefits that a better splitter alone cannot:
- Exact search on identifiers. Bid numbers, clause numbers and procedure codes become fields that lexical search can match exactly, which Part 2 showed is where most precision comes from.
- Filters. "Only current procedures", "only bids from the last five years", "only contracts we signed" become filters rather than hopes.
- Permissions. Each chunk carries the access rule of its source document, which is how Part 5 applies permissions before ranking.
- Honest citations. The answer can say "clause 14.3 of the BE-T-2024-0219 contract, revision 2" instead of "a document in the library".
Some teams also prepend a short context line to each chunk before embedding it, such as "From the BE-T-2024-0219 contract, clause 14, Delays and penalties". That helps semantic search place the chunk correctly, at the cost of slightly larger chunks. Test it on your own evaluation set rather than adopting it on faith.
Chunking meets document quality
Chunking cannot rescue content that was never machine-readable. Bergvliet's older bids included scanned PDFs with no text layer. They were "in the index" as chunks of nothing. Others had been exported from spreadsheets as images of tables. The photograph of a spreadsheet problem applies here exactly.
The team ran a simple census before re-chunking: for each source collection, how many documents had extractable text, how many had recognisable structure (headings, numbered clauses, tables), and how many were neither. The third group was sent back to the bid office with a choice: convert it properly, or leave it out of the Bid Library and say so in the assistant's scope. Leaving it out was often the right call. A document the assistant cannot read properly is safer absent than half-present.
Who decides
The platform team can build a structure-aware splitter. It cannot decide that a clause's exception must stay with its obligation, that a rate needs its validity date, or that meeting discussion is not worth indexing. Those are judgements about what the content means and how it is used.
At Bergvliet the decision sat with a small group: the knowledge management lead, one bid manager, one engineer and one platform engineer. They reviewed 20 sample documents per content type, agreed the rule and the metadata for each, and wrote the rules down as a short chunking specification. That specification is versioned. When it changes, the affected collections are re-indexed and the evaluation set is rerun, because a chunking change can quietly improve one kind of question and break another.
Key points
A chunk must be findable and self-sufficient. Fixed-size splitting serves neither on purpose.
Structure beats character counts. Split clauses by clause, procedures by step, tables with their headers.
Metadata does half the work. Identifiers, dates, versions, owners and permissions travel with every chunk.
Unreadable documents are safer absent. Census the corpus before you chunk it.
The business owns the rules. Write a versioned chunking specification and re-evaluate when it changes.
Next: Part 5, Permission-aware retrieval. All parts are on the series page.
How CloudNala can help
We help teams audit a corpus by content type, agree chunking and metadata rules with the people who own the content, and measure the effect on retrieval before and after. The result is a written chunking specification your platform team can maintain, rather than a magic number nobody remembers choosing.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a RAG Production Review or write to us at consult@cloudnala.co.za