Every promotion meeting has a slide with a rising line on it. Active users, sessions, prompts, documents processed. The line goes up and to the right, and the room relaxes. People are using it, so it must be working.
That inference is almost always too generous. A user count measures that people opened the system. It does not measure whether it changed how they work, whether they trust its output, whether anyone supports it, or whether the organisation is better off. In AI pilots, early usage is often inflated by novelty, by managers asking teams to try it, and by the simple fact that it is free to the user.
CloudNala's view: usage is the first step towards maturity, not evidence of it. And the people best placed to say whether a system is mature are rarely the people who built it.
Waterberg Mining Services is fictional, and its systems and figures are illustrative.
The staircase from curiosity to value
It helps to think of adoption as five steps, each harder to reach than the last.
Curiosity. People try it. This is what launch-week numbers measure.
Repeat use. The same people come back without being asked. Measure this with returning users per week, not total users.
Workflow fit. The system replaces a real step in someone's work. A supervisor no longer reads the full shift log because the handover note is reliable. A planner schedules maintenance from the model's list rather than from the old spreadsheet. This is where adoption starts to mean something.
Supported. Someone owns the system and fixes it when it breaks. Users can depend on it because there is a team behind it.
Measured value. Someone can show, with evidence, what has improved: time saved, errors avoided, incidents caught. Not a survey estimate. An observed change against a baseline.
In Part 1 we noted that three of Waterberg's Experiments had more than a hundred users each. All three were on the first or second step. None had reached the third with any evidence.
Why the team's confidence is not evidence
Project teams are usually right about their systems in a narrow sense. They know what it does when it works. What they cannot easily see is everything that happens around it: the user who quietly stopped trusting it, the support desk ticket that went nowhere, the workaround a site invented to cope with a gap.
This is not a criticism of teams. It is a structural blind spot. The CNCF recognises it in its own lifecycle, which requires interviews with adopters (five to seven of them) as part of due diligence before an open-source project moves up a level. The principle carries over well to an enterprise: ask the people who live with the system, not the people who built it.
Who to ask, and what only they can confirm
Frequent users can tell you whether it is useful and, more importantly, what it gets wrong. Ask for specific examples of mistakes, not a satisfaction score.
The workflow owner, the person accountable for the process the system sits in, can tell you whether the process actually changed. If the old steps are still being done "just in case", the system has not been adopted. It has been added.
The support team can tell you what breaks and how often. If the support team has never heard of the system, that is evidence too.
The security reviewer can tell you which findings remain open and whether the data the system touches matches what was approved.
The operations owner can tell you whether anyone could run it at two in the morning without the original developer.
Non-users are the group most often skipped and frequently the most informative. Ask people who tried it and stopped, or who were expected to use it and never started. Their reasons often reveal the real limits on value.
A short interview guide
Adopter interviews do not need to be long. Twenty minutes each, with five to eight people across those groups, gives a reviewer more useful evidence than any dashboard. Waterberg uses six questions:
- What do you use it for, specifically, in the last week?
- What did you do before it existed, and do you still do that?
- Tell me about the last time it was wrong. What happened next?
- If it disappeared tomorrow, what would you do?
- Who would you contact if it stopped working?
- Is there anything you will not use it for? Why?
Question 4 is the most revealing. "I would go back to reading the logs, it would cost me forty minutes a shift" is evidence of workflow fit. "I would not notice much" is evidence of curiosity.
Question 6 often surfaces trust boundaries the team never designed for. At Waterberg, several supervisors said they would never use the handover assistant to decide whether an area was safe to enter. That is exactly the right instinct, and it belongs in the system's documented limits rather than in individual supervisors' heads.
What the interviews found
Waterberg's reviewer interviewed seven people about the shift handover assistant: three frequent users, one non-user, the operations manager for two sites, the service desk lead and the information security officer.
The picture was mixed, and more useful because of it. Frequent users relied on the note for equipment status and blasting changes and said it saved real time. Two had seen it miss an open safety observation when the log was written in a mix of English and Sepedi. The non-user had stopped after a note summarised the wrong shift. The service desk had logged four tickets and had no idea who to route them to. The information security officer had not known the system existed until the review.
None of that appeared in the user count. All of it was essential to the decision. The assistant stayed an Experiment, with a new blocker added to its list: a test set of mixed-language shift logs, with a pass threshold agreed by the operations manager, before any promotion request.
Measuring adoption without vanity metrics
If you do want numbers alongside the interviews, choose ones that map to the staircase rather than to raw volume:
| Step | Better measure | Weaker measure |
|---|---|---|
| Repeat use | Returning users per week as a share of the intended group | Total registered users |
| Workflow fit | Share of shifts where the old manual step was skipped | Number of notes generated |
| Supported | Tickets resolved by the owning team, and time to resolve | Uptime alone |
| Measured value | Observed change against a baseline, with the method stated | Survey estimate of time saved |
For a broader treatment of the difference between activity and value, see the Boardroom AI Test series, which looks at how leadership should read these measures.
Next: Part 5, Due diligence belongs before promotion, not after failure. All parts are on the series page.
How CloudNala can help
We conduct independent adopter interviews and adoption reviews for AI pilots and internal platforms, so promotion decisions rest on evidence from the people who live with the system. We also help teams choose adoption measures that reflect workflow change rather than activity.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Technology and AI Maturity Assessment or write to us at consult@cloudnala.co.za