Why document AI creates value before conversational AI

Swartland Mutual Bank, the fictional South African bank in this series, had two AI proposals for its debt-review operations on the table at the same time. One was a chat assistant that would let customers in debt review ask about their status. The other was document AI for the inbox that 14 indexers were working through by hand.

The chat assistant had the better demonstration. The document AI had the better case, and the bank funded it first. The reason was not that conversation is unimportant. It was that the assistant would have answered questions from case data that was, on a typical day, three working days out of date.

This part explains why document AI so often creates value before conversational AI, what document AI actually consists of, and how to set it up so that its accuracy can be measured from the first week.

Documents have answers; conversations have possibilities

A Form 17.1 notice either names a particular customer or it does not. The proposed monthly instalment on a restructuring proposal is a specific amount. The date on a court order is one date. Every field the bank extracts from a document has a right answer that can be checked.

A conversation is different. A customer asking "where is my application?" may be asking about the debt review, a separate account, a payment they expected or a letter they received. There are many acceptable replies and some harmful ones, and judging which is which takes a rubric, as covered in A rubric is a specification for judgement.

That difference shapes everything that follows.

• Evaluation is cheaper. You can build a labelled set of documents with known correct values and measure the system field by field. Evaluate each AI layer differently explains why extraction can be tested this directly while generation cannot.
• History is a starting point. Swartland's indexers had keyed years of documents. That history is not perfect, because people make keying errors too, but a reviewed sample of it becomes a first evaluation set.
• Failure is visible. A wrong account number fails a cross-check. A wrong tone in a reply often fails silently.
• The cost of a mistake can be bounded. A document system that is unsure can stop and ask a person. A conversation that has already been sent cannot be recalled.

The five jobs inside document AI

"Document AI" is often described as one capability. In practice it is five jobs, and each needs its own design and its own measure.

1. Classification. Decide what the item is. At Swartland the categories include Form 17.1 notices, restructuring proposals, court and tribunal orders, termination notices, consumer letters, payment queries and mail that has nothing to do with debt review. An email with three attachments may hold three different documents, so classification works per document, not per email.
2. Extraction. Read the fields each type carries: identity number, account numbers, the debt counsellor's registration number, dates, proposed instalments. The fields depend on the type, which is why classification comes first.
3. Validation. Check every extracted value against something the bank already trusts. This is where most of the value hides, and it is covered in the next section.
4. Indexing. File the document against the right customer and accounts, and start the correct workflow. A notice starts a different workflow from a proposal.
5. Exception handling. Send anything the system cannot settle to a person, with the reason attached, and take it back once resolved.

Teams often build the first two jobs carefully and treat the last as an afterthought. That is backwards. The exception queue is where the hard cases go, where the people are, and where most of the process time ends up. Part 3 shows what happens when it is under-designed.

Validation is where most of the value hides

A model that reads an account number with high confidence can still be wrong. A model that reads it and then finds that the number exists, belongs to the customer named on the notice and is a credit product that can be under debt review has been checked by something other than itself.

Swartland's validation layer runs deterministic checks, not more model calls:

• The identity number has the expected format and matches a customer in the core banking system.
• Each account number exists, belongs to that customer and is of a type the process covers.
• The debt counsellor's registration number appears in the register of debt counsellors that the National Credit Regulator publishes (checked on 7 November 2026).
• Amounts on a proposal add up, and the instalment schedule is internally consistent.
• Dates are plausible: not in the future, and not before the account was opened.

These checks are cheap, explainable and stable, which is the argument of Use a workflow until the unknown forces you to use an agent: let rules do the work that rules can do, and confine the model to the step that needs judgement. Here the model reads the page; the rules decide whether what it read can be trusted.

Confidence is set per field, not per document

Many products return one confidence score for a whole document. That is rarely the right unit. A proposal can be read perfectly except for one smudged digit in the instalment, and a single document-level score either sends the whole thing to a person or accepts the smudge.

Swartland sets an auto-accept bar for each field and pairs it with an independent check.

Two design rules followed from that table.

• The most consequential field is never accepted on confidence alone. Filing a notice against the wrong customer is the costliest error in the process, so account matching always needs the core-banking check to pass, however confident the model is.
• A person sees only the fields that need them. If one field falls below its bar, the reviewer's screen shows that field, the image region it came from and the value the model proposed. The rest of the document is already settled.

The thresholds in the table are illustrative. Setting real ones is an evidence exercise, not a guess, and part 4 describes how to do it and how to keep checking them. Thresholds also belong to the action they unlock, a point made in Thresholds, evals and human review.

Turnaround and accuracy are measurable from week one

Because documents have right answers, document AI can be measured properly from the start. Swartland agreed five measures before the pilot:

Two habits made the numbers credible. First, every measure was reported per document type, because an overall figure can hide a type that never works. Second, field accuracy was measured on a fresh sample of live documents each month, not only on the set used during selection. A system tuned on last year's layouts can quietly degrade when a large debt-counselling firm changes its template.

This is also where model choice becomes practical. With a labelled set in hand, Swartland could test a smaller, cheaper model against a larger one on the same documents, as recommended in Use the smallest model that reliably passes the evaluation.

Personal information is the default, not the exception

Every debt-review document carries personal information, and much of it relates to a person's financial position. Document AI does not change that, but it does create new places where the data goes: an OCR service, a model endpoint, a store of extracted values, a labelled evaluation set and the logs of each.

Swartland treated each of those as a processing activity in its own right, using the approach in Which data can AI touch? A traffic-light guide. The evaluation set was held under the same access controls as the source documents, and retained only as long as it was needed for measurement. This is general information, not legal advice. Your own obligations under the Protection of Personal Information Act depend on your process and should be confirmed with your legal and privacy advisers.

When conversational AI comes second

None of this argues against a chat assistant. It argues for sequence. A customer asking "has my bank received my debt counsellor's notice?" deserves an accurate answer, and that answer can only come from an indexed case. A conversational layer on top of a three-day backlog would have answered quickly and incorrectly.

Once Swartland's inbox was indexed within the day for most items, the chat assistant's proposal changed character. It no longer had to guess. It could quote a case status that the back office had just set, and hand over to a person with the case already open. The articles When a chatbot is the right answer and Make the bank machine-readable explore that next step.

A sequencing test

Key points

• Documents have right answers, so document AI can be evaluated field by field from the first week. Conversations have many acceptable answers and need rubrics.
• Document AI is five jobs: classification, extraction, validation, indexing and exception handling. Design the exception queue as carefully as the model.
• Most of the value hides in validation: deterministic checks against records the bank already trusts.
• Set confidence per field, pair each field with an independent check, and never accept the most consequential field on confidence alone.
• Conversational AI works better second, once the case data it quotes is current.

This is part 2 of AI That Changes the Work, Not Just the Interface. Previous: The most valuable AI in banking may be the AI nobody talks to. Next: The AI is only one step in the process.

How CloudNala can help

CloudNala designs document AI for operational teams that need it to be measurable and defensible. We help you define the document types and fields, build a labelled evaluation set from your own history, design the validation checks and per-field thresholds, and specify the exception queue and reviewer screen, so that the system's accuracy is known before it handles live work and stays known afterwards.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Operational AI Review or write to us at consult@cloudnala.co.za