There is a slightly apologetic way people describe human review in AI projects. For now a person checks the output. Initially there is an approval step. The implication is that review is a stage of immaturity, and that the destination is a system nobody looks at.
For a small number of low-stakes, high-volume tasks, that destination is real. For most business workflows worth automating, it is not, and treating review as a temporary embarrassment causes two concrete problems: it gets designed badly because nobody expects it to last, and the information it generates gets thrown away.
The junior analyst comparison
A junior analyst produces work. A senior person marks it up. The junior improves, quickly at first, then more slowly, and after a year they are producing work that mostly does not need marking up. Nobody describes the senior's review as a bottleneck in that process. It is understood to be the process.
An agent improves in a similar operational sense, though the mechanism is different. It is not learning in the model-weights sense from each correction. What happens is that the correction feeds the parts of the system that are under your control: the example library gets a new corrected pair, the memory rules get an exception, the eval set gets a new case that will catch this class of error before the next release.
That is a real improvement loop. It is just an engineering loop rather than a training loop, and it only works if the corrections are captured in a usable form.
A rejection without a reason is a wasted rejection
This is the most common design failure in the whole area, and it is entirely fixable.
A thumbs-down tells you that something was wrong. It does not tell you what to change, so nobody changes anything, so the same error recurs, so reviewers conclude that giving feedback is pointless and stop doing it carefully. Every team that has run a feedback button for six months recognises this arc.
A small fixed set of reason codes changes the economics completely. Wrong source used, requirement missed, wrong output format, overstated certainty, outside the scope. Five options, one click, and each one routes to a different part of the system. Format problems go to the example library. Scope problems go to the memory rules. Sourcing and certainty problems go to the eval suite, because those are the failures you want caught automatically next time.
Keep the list short. A dropdown with fifteen options gets used as a dropdown with two options, and the two will be whichever appear first.
Three places this runs
Proposal and architecture packs. An architect approves the pack, corrects the assumptions, and rejects the architecture claims that are not supportable. That last category is the valuable one. "This claim is not defensible in front of a client" is expert knowledge that exists nowhere else and is exactly what the eval set should encode.
Tender compliance. A bid lead approves the extracted mandatory requirements and marks the ones that were missed. A missed requirement is the highest-cost error in the workflow, so each one found in review is worth adding to the eval set immediately, with the pack that produced it.
Lead triage. The user marks whether the suggested next action was useful. This is lighter-weight feedback and it is fine for it to be, because the cost of a wrong suggestion here is low. Match the weight of the review to the cost of the error, not to a uniform policy.
Designing review so it survives contact with a busy week
Review is a real cost, and pretending otherwise is how it gets dropped.
Put the review where the work already is. A reviewer who has to open a separate tool to approve something will approve it in batches, quickly, without reading. That is worse than no review, because it produces a signed-off audit trail attached to work nobody looked at.
Show the reasoning, not just the answer. A reviewer can check "deadline: 14 March, from clause 4.2 on page 6" in seconds. Checking a bare date requires them to redo the work, so they will either skip it or resent it.
Make partial approval possible. Most outputs are mostly right. If the only options are approve and reject, reviewers approve things containing one error they intend to fix later, and the error enters the record.
Route corrections automatically. If turning a correction into a new example is a separate manual task, it will not happen after the second week.
The part that does not go away
There is a category of judgement that stays with a person permanently, and it is worth being clear about which.
An agent can draft a bid or no-bid recommendation. It cannot be the party that decided to commit three weeks of the firm's capacity. An agent can produce a compliance matrix. It cannot be the party that certified the submission as complete. The accountability does not transfer, and the review step is where the accountability is exercised.
This is not a limitation to be engineered away. It is the reason the workflow can be used at all in a setting where being wrong has consequences. We have written more about this in why engineering judgement becomes more valuable.
What good looks like
Review designed as a permanent part of the system, not scaffolding.
A short fixed set of rejection reasons, each routing somewhere specific.
Reasoning and sources shown alongside the answer, so checking is fast.
Partial approval, so that one error does not force a binary choice.
Automatic capture of corrections into examples and evals, with no extra step for the reviewer.
Practical checklist
- Decide which decisions will always need a person, and say so plainly
- Add five reason codes to the review step, no more
- Route each reason code to examples, memory rules or evals
- Show the source and the reasoning next to every output
- Allow approval of part of an output
- Measure how long review takes, and treat a rise as a signal
How CloudNala can help
The review step is usually the cheapest thing to improve and the last thing anyone looks at. We tend to start by adding reason codes and routing, because that converts feedback people are already giving into data the system can use. After that the work is making review fast enough to survive a busy week: reasoning shown inline, partial approval, and corrections captured without the reviewer doing anything extra.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za