Correcting the knowledge base without preserving the error

Once the Blouberg Infrastructure Programme, a fictional public infrastructure programme with about forty construction sites, had traced the wrong "verified" answer about Site 7 back to a dropped word in a scanned inspection report, the obvious fix was to delete the bad fact. Sipho Ndaba had the query ready. Lindiwe Mokoena asked him to wait. (The programme's people, systems and figures are illustrative.)

Her reasons were practical. The wrong fact had been in the relationship store for nine weeks. In that time it had been summarised into two monthly progress reports, cached in an answer to a common question, used as expected evidence in an evaluation case, and quoted by a regional manager in an email that informed a payment certificate. Deleting the fact would remove one copy and leave the others. Worse, it would remove the evidence of what the assistant had believed in August, which internal audit had just asked for.

Correcting a knowledge base is not the same as editing a record. The error has usually travelled, and the history of the error is itself a record the organisation may need to keep.

Four kinds of change, four kinds of handling

The first step is to name what kind of change you are making, because each needs different handling.

Correction. The fact was wrong when it was recorded. The Site 7 verification is a correction: the source always said "not verified".
Supersession. The fact was right, and has been replaced by a newer one. A variation order that changes a milestone's scope supersedes the original; the original was not an error.
Retraction. The item should never have been in the knowledge base: a draft indexed by mistake, a document from the wrong programme, content the assistant was not approved to use.
Deletion. The item must be removed, for example because a retention period has ended or a person's information must be erased.

Operating RAG after launch covers the deletion checklist for withdrawn documents: indexes, caches and evaluation cases. This article is about the harder case, where the error is not in a document but in knowledge derived from it, and where the organisation must both fix the error and preserve its history.

Where an error lives

Using the provenance chain from Part 5, the team ran a forward query from the faulty extraction run. The result looked like this.

The source report was correct. The error entered when text recognition dropped a word and extraction turned the damaged sentence into a fact, which was then copied into rules, summaries, a cache, a test case and answers that informed a payment decision. Deleting the fact alone would have reached one of these places and erased the history auditors needed.

The origin. Not the source, which was correct, but the text recognition step that dropped "not", and the extraction step that turned damaged text into a fact.
Derived knowledge. The verification fact itself, and a relationship-store rule that had marked milestone 3 as "ready for certification" because of it.
Generated content. Two monthly summaries that repeated the fact.
Caches and tests. A cached answer and an evaluation case that expected the fault; Operating RAG after launch covers clearing these.
Answers already given. Fourteen answers to six users, one of which informed a payment decision.

A correction that fixes only the fact reaches one of these places.

Two timelines, not one

The reason not to delete is that a knowledge base answering questions for a public programme has to keep two timelines apart: what was true, and what the system recorded and said at the time. Database designers call this bitemporal data, with valid time and transaction (or system) time. The names matter less than the discipline.

The corrected fact at Blouberg is recorded as a new entity, "inspection 7-3-02 does not verify milestone 3", with a link to the faulty one. The faulty fact is not removed. It is marked as invalidated, with the time and the correction activity that invalidated it. The W3C's PROV-O vocabulary has terms for exactly this: an entity can be a revision of another (wasRevisionOf, "a derivation for which the resulting entity is a revised version of some original"), and it can be invalidated by an activity (wasInvalidatedBy, the "start of the destruction, cessation, or expiry of an existing entity by an activity").

The assistant reads only valid facts, the same precedence idea that Retrieval is not recall applies to agent memories. Auditors and investigators can still ask what the assistant knew on 14 August and receive the faulty fact, clearly marked as later corrected.

This is also the safer approach for a public body. Section 13(2)(a) of the National Archives and Records Service of South Africa Act 43 of 1996, as published in 1996, provides that no public record under the control of a governmental body shall be transferred to an archives repository, "destroyed, erased or otherwise disposed of without the written authorisation of the National Archivist". Whether a particular extracted fact or answer log is a public record is a question for the organisation's records manager and legal advisers; this is general information, not legal advice. An additive correction keeps that question open instead of answering it by accident.

The correction workflow

Blouberg now handles every reported error through the same sequence.

Every reported error follows the same sequence. The fix happens at the lowest layer where the error entered, forward provenance finds the derivatives, caches and people it reached, and the correction itself becomes a record. The faulty fact is kept and marked invalidated rather than deleted, so the system can still show what it believed and when.

Report and triage. Anyone can report a suspected error from the answer itself. A named owner classifies it (correction, supersession, retraction or deletion) and sets severity by what the error could cause: a misleading description is low; a wrong payment or safety statement is high.
Fix at the origin. Find the lowest layer where the error entered. If the source is wrong, the source owner corrects it. If a transformation was wrong, fix or re-run the transformation. Patching a downstream copy while the origin is wrong guarantees the error returns on the next rebuild.
Regenerate derivatives. Use forward provenance to find everything derived from the faulty item and rebuild it: facts, rules and summaries.
Invalidate serving copies. Clear cached answers that used the faulty item.
Add a regression case. Update the evaluation case that expected the error, and add one that asserts the corrected fact and the failure pattern.
Notify the affected. Use forward provenance to list everyone who received an answer that relied on the error. Tell them what was wrong, what is correct and whether any decision should be reviewed.
Close with a record. The correction itself is an entity, with who, what, when and why, linked to everything it changed.

At Blouberg, step 6 meant a short message to six people and a formal note to the contracts manager about the payment certificate issued in August. Payment against the certificate was withheld and the certificate reviewed pending the concrete retest. Without forward provenance, that certificate would not have been found.

When the error is about a person

Some errors concern personal information: a wrong qualification recorded against an inspector, a misattributed site diary entry. In South Africa, section 24 of the Protection of Personal Information Act 4 of 2013 lets a data subject request correction or deletion of personal information that is "inaccurate, irrelevant, excessive, out of date, incomplete, misleading or obtained unlawfully". Where the responsible party and the data subject cannot agree, and if the data subject so requests, section 24(2)(d) provides for attaching an indication that a correction was requested but not made, so that it will always be read with the information. Section 24(3) requires that, where a change affects decisions about the person, the responsible party must, if reasonably practicable, inform each person or body to whom the information was disclosed.

Two design consequences follow, offered as general information rather than legal advice. A knowledge base must be able to attach a note to a fact that travels with it into every answer. And forward provenance is what makes "inform each person to whom it was disclosed" practicable when the disclosure was made by an assistant.

Fix the cause, not only the instance

A dropped "not" is rarely a single event. The same text recognition engine, on the same kind of two-column scanned report, had processed about four hundred other inspection reports. The team quarantined every verification fact extracted from that combination, re-ran extraction with a layout-aware setting, and had inspectors confirm a sample. Nine further facts changed. None had yet reached a payment decision.

The general rule: when the origin of an error is a transformation, treat every output of that transformation, under the same conditions, as suspect until checked. Provenance makes the affected set a query rather than a guess. It is the same distinction drawn in An audit finding is not a corrective action: fixing the instance closes the finding, but only fixing the cause stops the next one, and the checks described in Data quality and observability are operational disciplines are how you notice when a transformation starts producing faults again.

Failure modes

Delete and forget. The fact disappears, the history goes with it, and the copies remain.
Fix downstream. A summary is edited while the faulty extraction still runs.
Silent correction. The knowledge base is fixed, and the people who acted on the error are never told.
One-off fixes for systematic faults. One instance is corrected while the transformation that caused it keeps producing more.

Key points

Classify every change as a correction, supersession, retraction or deletion, and handle each differently.
An error lives in derived facts, rules, summaries and answers already given, not only in one record.
Keep two timelines: correct by adding and invalidating, not by overwriting, so the system can still say what it knew and when.
Fix at the origin, regenerate derivatives, invalidate caches, add regression cases and notify everyone affected.
When a transformation caused the error, check every output it produced under the same conditions.

This is the final part of Enterprise Knowledge Architecture Beyond Basic RAG. For the retrieval foundations underneath it, see RAG That Survives Production.

How CloudNala can help

CloudNala helps organisations design correction into their knowledge architecture before the first serious error. We define the change types and owners, design additive correction and invalidation for your facts and answers, connect the workflow to your provenance records so affected answers and people can be found, and align retention and notification with your records management and POPIA obligations, together with your legal and records advisers.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book a Knowledge Architecture Review or write to us at consult@cloudnala.co.za