Before Ukhozi's gateway existed, each AI application held its own provider API key. The keys had been created by whoever built the application, stored in whatever configuration system that team used, and never rotated. When finance asked which application was responsible for a sudden jump on the provider invoice, nobody could say, because three applications shared one key.
The gateway fixed that by issuing virtual keys: credentials that the gateway itself recognises, each scoped to one application, while the real provider keys stay inside the gateway. It is one of the most practical features a model gateway offers. Open-source gateways such as LiteLLM use exactly this name, and let each key carry an allowed model list, a spend budget with a reset period, token and request rate limits, an expiry and a team for roll-up reporting.
CloudNala's view: a virtual key is an accountability mechanism. It tells you which application spent what, within which limits, and lets you stop it. It is not an identity model, and treating it as one creates a gap that looks like a control.
Ukhozi Telecom is fictional, and its systems and figures are illustrative. By this point Ukhozi had two applications behind its gateway, after the narrow proof described in Part 2.
What a key can answer
The left-hand column is where virtual keys shine:
- Which application called. One key per application per environment makes attribution trivial. Ukhozi's naming convention was
app-environment-purpose, for example the store-support assistant's production key. - Which models it may use. The HR policy assistant had no reason to call the most expensive model, so its key did not allow it.
- How much it may spend. A monthly budget per key, with alerts at 50% and 80% and a hard stop at 100% for non-critical applications.
- How fast it may call. Request and token rate limits protect the shared provider quota from one noisy neighbour.
- When the key expires. Keys that expire force rotation to happen, rather than being promised.
Those five answers cover most of what finance and the platform team need day to day.
What a key cannot answer
The right-hand column is where teams get into trouble.
Which person asked. A key identifies the application. The billing-dispute agent serves thousands of customers through one key. If the gateway log is the only record, it shows the application, not the customer or the staff member. For that, the application must pass a user or session reference through to the gateway (as metadata, not as the credential), and log it in its own trace.
Whether they may see the data. A key that allows a model says nothing about whether the prompt contains data the user is entitled to. That decision belongs in retrieval and in the source systems. Identity before intelligence covers the full identity chain from user to model.
Whether the action is allowed. A model call is not an action. When the agent then calls the billing system, the billing API must decide. A key that can reach a powerful model does not make the resulting action authorised.
Whether the caller is really the application. A key is a bearer secret: whoever holds it can use it. If possible, bind it to the workload, for example by allowing the key only from the application's network or by having the application authenticate to the gateway with a workload identity and receive a short-lived token. Where that is not possible, keep keys in a secret store and treat every copy as a risk.
Whether the key was copied. Ukhozi's review found the network-fault summariser's provider key in a shared notebook and a developer's local environment file. A gateway key copied the same way has the same problem, just with smaller blast radius.
Keys need a lifecycle
The other mistake is treating key issue as a one-off. A key has a lifecycle, and every step needs an owner.
Request. The application owner asks for a key, naming the purpose, the environment, the expected volume and the cost centre. No purpose, no key.
Scope. The platform team sets the allowed models and routes. Defaults are narrow: the models the application has been evaluated on, nothing else.
Budget and limits. The cost owner sets the monthly budget and decides what happens at the limit: alert only, throttle, or stop. Ukhozi chose "stop" for the HR assistant and "alert and throttle" for the billing-dispute agent, because stopping a customer-facing dispute flow mid-month was worse than an overspend conversation.
Monitor. Weekly review of spend and error rate per key. A key whose spend doubles without a change in usage is the earliest sign of a loop or a prompt change.
Rotate. On a schedule, and immediately when a person with access leaves or a copy is found. Rotation should be routine enough that nobody is nervous about it.
Revoke. Anyone on the platform on-call rota may block a key immediately, without asking. Restoring it needs the application owner. Ukhozi rehearsed this once a quarter: revoke a test key, confirm traffic stops within a minute, restore it.
Practical defaults
- One key per application per environment; never shared between applications.
- Production keys never on laptops; development keys with small budgets and short expiry.
- Every key tagged with an owner and cost centre, and reported weekly by that tag.
- Budget behaviour chosen per application, with the business owner's agreement.
- A user or session reference passed as metadata for traceability, never used as a credential.
- Revocation tested, not assumed.
The short version
Keys answer accounting questions. Which application, which models, how much, how fast, until when.
Keys do not answer identity questions. People, data entitlement and action authority live elsewhere.
Bind keys to workloads where you can. A bearer secret is only as safe as its storage.
Give every step of the lifecycle an owner. Especially revocation.
Previous: Part 2, one model, one provider, one consumer. Next: Part 4, who owns the retry? All parts are on the series page.
How CloudNala can help
We help platform teams design key schemes that finance can report on and security can live with: naming and scoping conventions, budget behaviour per application, workload binding where the platform allows it, user traceability without turning keys into identities, and a revocation drill that proves the off switch works.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za