Every capability programme eventually has to answer one question from the people funding it: is it working?
The easiest answers are about activity. So many people attended, so many completed the online modules, so many passed a vendor certification. These numbers are real and they are reassuring. They are also weak evidence of the thing the programme was meant to build, which is the ability of teams to deliver safely and well with agents.
Kalahari Assurance is fictional, and its systems and figures are illustrative. At the end of its first year, Kalahari could report that 350 people had attended the prompting workshop and 42 had passed a vendor AI certification. It could not say whether any team was better at delivering change with agents than before. When the programme was redesigned, the first decision was to change what counted as evidence.
A ladder of evidence
CloudNala uses a simple ladder to talk about evidence of capability. Each rung is easier to count than the one above it, which is why programmes tend to report the bottom of the ladder.
Attended a session. Shows exposure. Says nothing about skill.
Holds a certificate. Shows that someone learnt a syllabus, usually for one vendor's tools, at one point in time. Certifications have value for individuals and for some procurement requirements. They are also highly volatile: exams, names and content change as fast as the tools do. This article does not rank them, and neither should a capability programme.
Finished an exercise. Shows that someone can apply a technique in a controlled setting with a known answer.
Reviewed work product. Shows that someone produced a real artefact, such as a specification, an evaluation set, a harness configuration or a pull request, and that another qualified person reviewed it against a standard.
Operated a production change. Shows that someone took a change built with agents into production, with the evidence a production change requires, and that it held up.
Kalahari reports all five rungs, because the bottom ones are useful for planning. Only the top two count towards a role's capability record.
A production change is a better capstone than a certificate
The programme's capstone is not an exam. It is a small production change, built with agents, by a pair or a small team, inside the governed toolchain, and reviewed against six pieces of evidence.
Each line is deliberately something that cannot be produced by memorising a syllabus.
Specification. Every acceptance criterion maps to a test. This connects directly to Part 3.
Evaluation set. The evaluation includes the cases the system gets wrong, with a record of what was done about them. A capstone where everything passes is treated with suspicion, for the same reason a first AI project should include failure and cost.
Tool boundaries. The agent's permissions are least privilege and were reviewed by someone outside the team.
Cost per case. Measured, and inside the ceiling agreed before the work started.
Runbook. Rollback has been rehearsed at least once, not just written down.
Handover. A named operational owner has accepted the change. This is the line that most often fails on the first attempt, and the most important one.
Kalahari's capstones are real backlog items, chosen to be small and low risk: a claims document classifier that suggests rather than decides, a broker portal search improvement, a reconciliation report. The work ships. The evidence becomes part of the team's record and, because it uses the organisation's own standards, it doubles as a test of whether those standards are usable.
Measuring the organisation, not only the individual
Individual capability records help with staffing and development. Leaders also need to know whether the organisation is getting better. For that, Kalahari tracks a small set of measures taken from the work itself, each reviewed quarterly:
| Measure | What it tells you | Where it comes from |
|---|---|---|
| Changes with testable acceptance criteria | Specification literacy is spreading | Specification review records |
| Agent changes passing local checks before review | Harnesses and skills are maturing | Pipeline data |
| Review time per accepted change | The review bottleneck from Part 2 is easing | Pull request data |
| Escaped defects traced to misunderstood intent | Whether specifications capture what matters | Incident and defect records |
| AI assets with a named owner and review date | Ownership culture is holding | Registry and standards records |
| Cost per accepted change | Whether capability is efficient, not only fast | Cost attribution data |
Several of these overlap with the engineering metrics in four AI engineering metrics teams should measure every week. That is intentional. A capability programme should be judged by the same evidence the delivery system already produces, not by a separate set of training statistics.
Make evidence cheap to collect
Work-product evidence has one practical weakness: if it takes effort to gather, people stop gathering it, and the programme quietly falls back to counting attendance. The answer is to collect it as a by-product of the work rather than as a separate reporting task.
Kalahari's platform team did most of this with small changes to existing tools. The pull request template asks for a link to the specification and records whether its criteria map to tests. The pipeline records whether an agent-authored change passed local checks before a human reviewed it. The registry and the standards repository already hold owners and review dates. Cost attribution comes from the gateway. Only two measures need people: the quarterly quality sample and the capstone review.
The capability record itself is generated, not filled in. When a capstone passes review, the reviewer's sign-off adds an entry for each team member, linked to the change. When someone is the named owner of a standard or a registry entry, that appears too. A person's record is therefore a list of links to real work rather than a list of courses.
This has a useful side effect. Because the evidence comes from the same systems the team uses every day, gaps in it usually point to gaps in the delivery system itself, such as a team that does not use the pull request template, rather than to gaps in reporting.
Avoiding measurement theatre
Work-product measures can be gamed like any other. Three precautions help.
Measure trends, not targets. A target of "90 per cent of changes with acceptance criteria" invites criteria that exist only to tick the box. A rising trend with spot reviews of quality is harder to fake and more honest.
Sample the quality. Each quarter, a small group from outside the team reviews a random sample of specifications, evaluation sets and capstones against the standards. The sample is small. Its existence keeps the numbers honest.
Keep failures visible. A capstone that failed its handover review, or an evaluation set that revealed a serious weakness, is useful evidence. If failures disappear from the record, the record stops being trusted.
What this means for hiring and promotion
Once work-product evidence exists, it can inform people decisions. Kalahari now asks candidates for senior engineering and architecture roles to walk through a piece of their own agent-assisted work, using the same six lines as the capstone. The portfolio advice in the individual series describes what good looks like from the candidate's side. Internally, promotion to senior roles requires at least one operated production change in the capability record.
Next: Part 8, Four levels of AI engineering maturity, and a curriculum that keeps up. All parts are on the series page.
How CloudNala can help
We help organisations replace activity metrics with work-product evidence: an evidence ladder suited to their roles, a capstone design built around real low-risk backlog items, and a small quarterly measurement set drawn from data the delivery system already produces. We can also act as the independent reviewer for a first round of capstones.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Plan an AI Capability Programme or write to us at consult@cloudnala.co.za