The business at Komati Commerce, the fictional South African commerce platform in this series, wanted free delivery to depend on the region. Instead of a single threshold for the whole country, each region would get its own, with a lower one for metros and a higher one for remote areas. A year earlier, a coding agent would have opened the pricing engine, changed the threshold logic and written a test. This time, the first thing it produced was not code. It was a list of everything the change would touch, with the evidence for each item, and a person approved that list before a line was written.
The list included the invoice builder's copy of the fee rule and the reconciliation view, the two places that had caused the defect in part 1. It also included a partner feed, a margin alert and a runbook. When the change was finished, the pull request showed how the prediction compared with what had actually changed, including two things nobody had predicted.
This article sets out that practice: predict impact before generating code, then compare the prediction with the result and attach the comparison to the pull request as evidence.
Why impact comes first
There are three reasons to ask for impact before code.
A plan is cheaper to correct than a diff. If the predicted impact misses the invoice builder, a reviewer can add it in a minute. If the diff misses it, the defect is found by finance on a Friday.
It makes the agent use the map. An agent asked to write code will search until it finds somewhere to write. An agent asked to state impact has to traverse callers, data and tests first. The map from parts 1 to 3 is then a requirement of the task, not an optional extra.
It gives the reviewer something to check against. Without a prediction, a reviewer reads the diff and asks whether it looks right. With a prediction, they can ask whether the diff did what was planned, and why it did more or less than that.
This extends the specification work in Spec-Driven Development rather than replacing it. The specification is the control surface argued that intent should be written down before code. The impact map does the same for consequences.
Six kinds of impact
A change touches more than code. The impact map lists six kinds, each with its own source of evidence.
| Kind | Examples at Komati | Evidence |
|---|---|---|
| Code | Pricing engine, invoice builder, cart banner | Graph traversal from the changed symbols |
| Contracts | Quote API response, order.priced event, partner feed | API and event schemas, consumers listed in the catalogue |
| Data | regions table, reconciliation view, a migration | Schema, lexical sweep for column names |
| Tests | Pricing unit tests, a new invoice-parity test, the API contract test | Tests that reach the changed symbols, plus gaps |
| Operations | Feature flag for the rollout, margin alert, runbook | Alert definitions, flags, runbooks that name the component |
| People | Pricing owner, fulfilment owner, finance contact | Code owners and the module ownership register |
Each item is classed as will change, must verify (likely unaffected, but someone must confirm) or unaffected, with a reason given. The reasons matter more than the classes. "Partner feed: must verify, because it publishes delivery_fee but not the threshold" is a sentence a reviewer can check.
The change plan
The impact map sits inside a short change plan that the agent produces and a person approves. Komati Commerce's plan has a fixed shape, kept in the pull request description so that it travels with the change.
intent: Free-delivery threshold set per region instead of nationally
spec: PRICING-214 (acceptance criteria 1-4)
seeds: [PricingEngine.deliveryFee, regions]
traversal: callers and data, 2 hops; lexical sweep for delivery_fee, free_delivery
impact:
- item: InvoiceBuilder.recompute
class: will-change
reason: holds its own copy of the fee rule
- item: recon_daily_takings (SQL view)
class: must-verify
reason: reads orders.total_cents; fee copy removed after the earlier defect
- item: partner-feed
class: must-verify
reason: publishes delivery_fee, not the threshold
tests:
add: [invoice-parity, regional-threshold-boundaries]
run: [pricing-unit, order-api-contract]
rollout: behind flag free_delivery_by_region, metros first
unknowns: [delivery-slot service reads fees via a config key; not traced]
owners_to_notify: [pricing, fulfilment, finance]
Three fields deserve attention. traversal records how the impact was found, so a reviewer can judge how far to trust it. tests.add names the test the earlier defect showed was missing. unknowns is where the agent says what it could not trace, and here it named the service that later turned out to be affected.
Approving the plan, proportionately
Not every change needs a plan reviewed by a person. Komati Commerce sets the bar by impact, not by size.
- A change contained in one module, with no contract, data or operational impact. The agent writes the plan, and the reviewer reads it with the diff. No separate approval is needed.
- A change that crosses modules or touches a contract. The module owners on the
owners_to_notifylist approve the plan before code is generated. - A change that touches money, personal information or an external partner. The plan is approved by the owner and the accountable business contact, and the unknowns must be resolved or accepted by name.
For agents working unattended, the task classes in What the night shift is allowed to merge already decide which work may run without anyone awake. The impact map is how a task proves it belongs in its class. A change predicted to touch a contract is not a night-shift task, whatever its ticket says.
Predicted versus actual
Once the agent has written the code and the tests have run, the comparison is mechanical. Take the symbols, files, schemas and configuration in the actual diff, add anything the changed code now reaches in the rebuilt map, and set them against the prediction.
The result falls into three groups.
Predicted and changed. These items went as planned. Each one is linked to the test that exercises it, which turns the prediction into evidence.
Predicted, not changed. These were expected to change, or to need verification, and did not change. Each needs a one-line reason: "verified: feed publishes the fee, not the threshold" or "not needed: alert keys on margin, not fee". If there is no reason, the prediction was noise, and the map or the traversal depth should be tightened.
Changed, not predicted. This is the important group. At Komati, the agent also changed the delivery-slot service, which read the threshold through a configuration key, the unknown it had flagged, and a shared rounding helper. Both changes were correct, but neither had been reviewed as part of the plan. Each item in this group needs an explanation, a test and an owner's acknowledgement before merge.
The size of the last group is the single best signal of how well the map and the plan reflect the system. An empty group on a cross-module change is worth a second look, while a large one means the map is missing edges.
The comparison as pull-request evidence
CloudNala has written about what evidence should accompany an agent's change, and this article does not repeat it. Let the agents take the night shift set out the evidence pack that decides what is merged in the morning. From prompt to change record connected the request, the plan, the approval and the result into one record. AI code review is a filter, not a sign-off set out where automated review fits.
The predicted-versus-actual comparison is the impact section of that evidence pack. It adds three things the pack did not have:
- A prediction made before the code existed, so the agent's understanding of the system can be checked independently of its code.
- An explicit list of surprises, each with an explanation, a test and an owner's acknowledgement.
- A link from each impacted item to test evidence, so a reviewer can see which predictions were proven and which were only asserted.
Komati Commerce's merge rule is simple. A cross-module change does not merge while the "changed, not predicted" group contains an item without an explanation, a test and an owner's acknowledgement.
Learning from the gap
Every surprise is a fact about the system that the map did not know. Komati Commerce routes each one back.
- A configuration-key dependency becomes an entry in the string-sweep list from part 3, so the next traversal finds it.
- A missing test becomes a test, which is what happened to invoice parity.
- A link that surprised the reviewer becomes a
WHYcomment or a short decision record, the gate from part 4. - A recurring surprise in one module is a sign of the hidden coupling that part 4 described, and a candidate for a fitness function.
Over a quarter, the team watched the surprise count on cross-module changes fall. Its figures are its own and illustrative. The trend matters more than any target.
A checklist
| Before code | After code | At merge |
|---|---|---|
| Intent and spec reference stated | Actual diff and rebuilt map compared with the prediction | Every surprise explained, tested and acknowledged |
| Seeds and traversal recorded | Each predicted item linked to a test result | Predicted-not-changed items have a reason |
| Six kinds of impact listed with reasons | Unknowns resolved or accepted by name | Comparison attached to the evidence pack |
| Tests to add and run named | Surprises routed back to the map | Owners on the plan have approved |
| Owners notified; plan approved where impact requires it |
Key points
- Ask the agent to predict impact before it generates code. A plan is cheaper to correct than a diff, and the prediction forces the agent to use the map.
- List six kinds of impact (code, contracts, data, tests, operations and people), each item with a class and a reason a reviewer can check.
- Approve plans in proportion to impact. A change predicted to touch a contract or money is not a routine task.
- After the code, compare predicted with actual. Changes nobody predicted are the important group, and each needs an explanation, a test and an owner.
- Attach the comparison to the pull request as the impact section of the existing evidence pack, and feed every surprise back into the map.
This is part 5, the final part, of Codebase Intelligence for AI-Assisted Engineering. Previous: The code works. Nobody understands the system. To start the series again, read Your coding agent does not need more context. It needs a map. For the controls around agents more broadly, see The Coding Agent Harness.
How CloudNala can help
CloudNala helps teams introduce predicted impact maps and predicted-versus-actual comparisons into the way their coding agents work. We define the change-plan format and approval thresholds for your repositories, connect the plan to your code map, tests and ownership register, and add the impact section to your pull-request evidence, so reviewers can see what an agent expected to change, what it actually changed and why the two differ.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Codebase Intelligence Review or write to us at consult@cloudnala.co.za