"It is just a pilot" is one of the most expensive sentences in technology. It is usually said to lower the bar: for security review, for data approval, for support planning, for cost tracking. That is reasonable for something that is genuinely small, short and contained. It becomes a problem when the word stays the same while the system grows around it.
In Part 1 we introduced a four-stage lifecycle: Experiment, Incubating, Trusted production, and Retiring or archived. The first stage is the one organisations find easiest to name and hardest to keep honest. This part argues that an experimental label is a risk label. Its job is to warn anyone who encounters the system about what it can and cannot be relied on for. If the label does not carry that warning, it is not a label. It is a place to hide a permanent system from the scrutiny a permanent system needs.
Waterberg Mining Services is fictional, and its systems and figures are illustrative.
How a pilot promotes itself
Waterberg's shift handover assistant is a good example of a system nobody promoted. Two engineers built it in their spare time to summarise the free-text shift logs that supervisors write at the end of each shift at client mines. It used a commercial model through an API key on one engineer's company card. The output was a short handover note: open safety observations, equipment out of service, blasting schedule changes, and anything the incoming supervisor should check first.
It was good. Within three months, 140 supervisors at seven sites were using it, and night shift had started relying on it to catch up on what day shift had done. In month seven the API key expired over a long weekend. The handover notes stopped. Nobody knew who to call, because nobody had been asked to support it.
When the incident review looked closer, it found more than a missing key. The shift logs contained employee names, injury details from safety observations and occasional client commercial information. None of that had been approved for processing by an external model. The monthly cost had grown from a few hundred rand to an amount someone in finance had queried twice without getting an answer. The prompt had been edited several times, with no record of which version had produced which note.
None of this was negligence. It was the natural result of a system that was still called a pilot while it carried the weight of production.
What the label must say
An experimental label should answer six questions for anyone who finds the system, whether that is a new user, an auditor, a security analyst or a supplier:
Data. Which data is allowed? For most Experiments the honest answer is synthetic data, public data or a small approved sample. If the experiment needs real personal information, that is a decision for the data owner and privacy officer, recorded before the data flows, not after.
Users. Who may use it? Named testers, not anyone with the link. An Experiment with open access is no longer testing a hypothesis. It is running an unsupported service.
Support. What happens when it breaks? "Best effort during office hours, no on-call" is a perfectly good answer for an Experiment, as long as users know it before they depend on the system.
Change. How stable is it? Experiments should expect breaking changes to prompts, interfaces and outputs. If other systems already depend on its output format, it has outgrown the label.
Spend. What is the cost ceiling? A hard monthly limit set at the provider or gateway, with a named person who is alerted when it is approached. AI services priced per use can grow quietly. If you are unsure how that happens, Why AI is an operating cost explains the mechanics.
Expiry. When will someone decide? Every Experiment should have a review date. At that date it is promoted to the Incubating gate, extended once with a reason, or stopped.
Waterberg now publishes these six answers on a single page linked from every Experiment's landing screen and from the portfolio list. It takes a sponsor about twenty minutes to complete.
Restricting an Experiment without killing it
The fear behind loose sandbox labels is usually that controls will smother useful experimentation. That fear is fair, and the answer is to make the restrictions cheap rather than to skip them.
Some practical patterns:
- A shared experimentation environment with pre-approved model access through a gateway, per-experiment budgets and logging already switched on. Teams start inside it by default rather than on personal keys.
- A standing approval for low-risk data, such as public documents, synthetic records and anonymised samples, so most Experiments never need a bespoke privacy review.
- Automatic expiry of access after the review date unless someone renews it with a reason.
- A visible banner in the interface: "Experiment. Not supported. Do not use for decisions about people or money." This sounds trivial. It changes behaviour.
These are not heavy controls. They are what turns "pilot" from a reassurance into a warning.
When the label is wrong
The shift handover assistant had been called a pilot for seven months while behaving like production. Waterberg's lifecycle review had three options.
It could stop the assistant, which would have been defensible on data grounds but would have removed something supervisors valued.
It could promote it straight to Incubating, which would have rewarded the very pattern the review was trying to stop, with known blockers still open.
It chose the third option: keep it labelled Experiment, freeze its user base at the existing 140 supervisors, and give it a short list of blockers to close before its first gate in eight weeks. The blockers were: a named product owner from operations, a data approval covering shift logs (with names and injury details stripped before the model call), a cost ceiling on a company-managed key behind the gateway, versioned prompts, and a support arrangement with the service desk.
Freezing growth is underrated. It respects the value users already get while preventing the risk from growing until the evidence exists.
The signs a sandbox is hiding something
You can usually spot a permanent system wearing an experimental label by looking for any of these:
- Other systems consume its output on a schedule.
- People outside the original team ask for access, and get it.
- It has been an Experiment for longer than its review date, and nobody noticed.
- Its costs appear in a general cloud or software budget line rather than against a sponsor.
- An incident would cause a business process to stop, not just inconvenience the team.
Any one of these is a signal to schedule a gate. Two or more usually means the gate is overdue.
For a related view from the delivery side, The demo is the beginning looks at why a successful demonstration is a starting point for engineering work rather than a finish line.
Next: Part 3, What does production-ready actually mean?. All parts are on the series page.
How CloudNala can help
We help organisations set up experimentation environments that are cheap to use and safe by default: gateway-mediated model access, per-experiment budgets, pre-approved data tiers and simple expiry rules. We can also review your current pilots against the six questions above and tell you which ones have quietly become production services.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Technology and AI Maturity Assessment or write to us at consult@cloudnala.co.za