Data lineage for AI answers

In October, the internal audit unit at the Blouberg Infrastructure Programme, a fictional public infrastructure programme with about forty construction sites, sent Lindiwe Mokoena a short question. In August, the programme's assistant had told a regional manager that milestone 3 on Site 7 had been verified by inspection. A payment certificate was issued the following week. The inspection report, read by a person, said the opposite: the works had not been verified, pending a retest of concrete cubes. On what basis had the assistant said otherwise? (The programme's people, systems and figures are illustrative.)

The team had a log. It showed the question, the user, the answer and the identifiers of the items retrieved. One of those items was a fact in the programme's relationship store: "inspection 7-3-02 verifies milestone 3". The log could not say where that fact had come from. It took Sipho Ndaba two days to reconstruct the chain: the inspection report was a scanned PDF; text recognition had dropped the word "not" from a line broken across two columns; an extraction job had turned the damaged sentence into a verification fact; and the assistant had faithfully reported it.

Then internal audit asked the second question, which was harder. Who else had been told something that depended on that fact? The log could answer questions forwards in time from a question, but not backwards from a source. Nobody knew.

Pipeline lineage and answer provenance are different

Data teams already know about lineage. Data quality and observability are operational disciplines treats it as a way to trace any signal in a dataset to its upstream cause and downstream impact. That lineage usually works at the level of datasets and jobs: this table was built from those tables by that pipeline.

An AI answer needs something finer. It needs to know which specific records, at which versions, passed through which specific transformations, to support which specific statements, for which user. Call this answer provenance. The retrieval pipeline's evidence log, described in The retrieval pipeline, end to end, records the last step: what was retrieved and cited. Answer provenance extends that record back through every transformation to the original source, and makes it searchable in both directions:

Backwards: why did the assistant say this? Which sources, which versions, which extraction?
Forwards: if this source, or this extraction run, was wrong, which answers and which people did it reach?

Blouberg's log supported half of the first question and none of the second.

A vocabulary that already exists

There is no need to invent a model for this. The W3C's PROV family of documents, published in April 2013, defines provenance as "information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness" (PROV Overview). The PROV-O Recommendation gives three core types and a handful of relationships:

an entity is a thing with fixed aspects: a scanned report, an extracted fact, an index version, an answer;
an activity is something that happens over time and acts on entities: text recognition, extraction, retrieval, generation;
an agent bears responsibility for an activity: an inspector, an extraction service, a model, a user;
an activity used entities, and an entity was generated by an activity;
an entity was derived from another, and was attributed to an agent.

You do not have to store PROV as RDF. The concepts work just as well as columns in ordinary tables. Their value is that they force the right questions: what was this generated by, from what, and under whose responsibility?

The answer provenance chain

Applied to the Site 7 answer, the chain as the programme now records it looks like this. (The statement support check was added after the incident.)

The chain as Blouberg now records it; the statement support check was added after the August incident. Each pipeline stage records what it used, what it generated and which agent and version was responsible, in the W3C PROV terms of entities, activities and agents. With every link recorded, the chain can be read backwards to explain an answer and forwards to find every answer and person a faulty extraction reached.

Every arrow is a record the programme now keeps. The scanned report is an entity with a version. Text recognition is an activity that used it, with the engine and its version recorded. The recognised text is an entity generated by that activity. Extraction is an activity that used the text and generated facts, each with the extraction job's identifier and the page and line it came from. Retrieval is an activity that used the index and the user's permissions. Generation is an activity that used the retrieved items, a model version and a prompt version, and generated an answer. The answer is attributed to the assistant on behalf of the user who asked, and each statement in it points to the entities that support it.

With that chain, the audit question becomes a query. Backwards from the August answer, the chain leads to extraction run 2026-08-03 and page 4, line 17 of the scanned report. Forwards from that extraction run, it leads to every fact the run produced, and from those facts to every answer that used them.

Statements, not just answers

Linking an answer to a list of sources is useful but blunt. A three-line answer may draw each line from a different place, and a reviewer needs to know which. Blouberg's answers are now produced as a small set of statements, each with its own evidence identifiers, and a checker confirms before the answer is shown that every statement has at least one supporting entity the user was permitted to see. Statements without support are removed or marked as unsupported.

This also makes the answer contracts from Part 1 enforceable. If the contract says payment status must come from the finance system, the checker can confirm that the payment statement's evidence is a finance record, not a progress report.

Record at a depth that matches the risk

Full provenance for every answer is expensive to store and, because questions and answers often contain personal information, creates a new store that must itself be governed. Do your observability tools store the prompt? covers that risk. Blouberg chose three tiers.

Full provenance for every answer is costly and creates a new store of personal information, so depth follows the risk of the answer. Tier 2 is the level that lets an audit question about a payment or verification be answered in minutes, and the answer contract, not the user, decides which tier applies.

Tier 1, every answer. Question identifier, user and role, answer identifier and time, source identifiers with versions, model and prompt versions, and the ontology version from Part 4. Enough to answer "which sources" and to run a forward query by source.

Tier 2, answers that state facts about contracts, payments or verification. Everything in tier 1, plus the identifiers of the extraction and indexing runs behind each item, the permission decision, and the statement-to-evidence mapping. Enough to answer the audit question in minutes rather than days.

Tier 3, answers used in a formal decision. Everything in tier 2, plus a fingerprint of the evidence text as shown, the reviewer who accepted the answer and the decision it informed, retained for the same period as the decision record under the programme's records management rules.

The tier is set by the answer contract, not by the user. Most questions are tier 1. The questions that lead to money moving are tier 2 at least.

What Blouberg built

The implementation was modest. Each pipeline stage writes a provenance row when it runs: what it used, what it generated, which agent and version, and when. Facts in the relationship store carry the identifier of the extraction run that produced them. The answer service writes a row per answer and per statement. All of it lands in the programme's existing data warehouse, where both directions are ordinary queries.

Three practical lessons came out of building it:

Identifiers must be stable from the start. A provenance chain is only as good as its weakest join. Extraction runs, index versions and documents all need identifiers that never change meaning, the same rule the identifier register follows for contractors and sites.
Record at the time, not afterwards. Reconstructing provenance from logs after an incident is slow and incomplete. Each stage must write its record as part of its own work.
Generated content is an answer too. The programme had started using the assistant to draft sections of its monthly progress summaries. Those drafts were edited by people and circulated widely, and at first they carried no provenance at all. They now record the same tier 2 information as an answer, plus the editor who approved the final text, because a summary that repeats a faulty fact spreads it further than any single answer.

None of this needed a specialised lineage product. The concepts mattered more than the tooling, and the warehouse the programme already ran was enough to hold a few million rows a year. Organisations that already operate a dataset lineage tool can extend it with answer-level records; the important thing is that both directions of the query work across the boundary between data pipelines and the assistant.

A checklist

Give every source version, transformation run, index build, model version and answer a stable identifier.
Make every pipeline stage record what it used and generated, by whom, and when.
Carry extraction run identifiers on every derived fact and chunk.
Produce answers as statements with their own evidence, and check support before display.
Set provenance depth by answer contract, and govern the provenance store as a store of personal information.
Test both directions: pick a past answer and trace it to its sources, then pick a source and list every answer it reached.
Include provenance checks in evaluation: an answer without traceable support is a failure, even if it is correct.

Key points

Dataset lineage is not enough for AI answers. Answer provenance records which specific records and transformations supported which statements for which user.
Provenance must work in both directions: backwards to explain an answer, forwards to find everyone affected by a bad source.
W3C PROV's entities, activities and agents give a ready vocabulary; you do not need RDF to use it.
Map evidence to individual statements and check support before an answer is shown.
Record at a depth that matches the risk, and govern the provenance store itself.

Next: Part 6, Correcting the knowledge base without preserving the error. All parts are on the series page.

How CloudNala can help

CloudNala designs answer provenance for assistants that must stand up to audit. We map your pipeline's transformations, define stable identifiers and the provenance record each stage writes, set provenance tiers against your answer contracts and records obligations, and run a trace exercise in both directions on real answers, so that the next time someone asks why the assistant said something, the answer takes minutes and is complete.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book a Knowledge Architecture Review or write to us at consult@cloudnala.co.za