Self-hosting an open-source model gateway takes an afternoon. A container, a database, a configuration file and two provider keys, and there is an admin screen showing models, keys and a pleasing spend chart. That afternoon is often presented as "we have an AI gateway".
What Ukhozi actually had, at that point, was a demo. Three months later it was a shared service on the critical path of five applications, including the billing-dispute agent customers used directly. If it went down, five applications went with it. If its database was lost, so were every key, budget and routing rule. If it fell two versions behind, the next security fix would arrive with three breaking changes attached.
CloudNala's position: a gateway becomes a platform when someone operates it, not when it is installed. Operating it is ordinary infrastructure work with named owners, and it should be budgeted before the second application joins.
Ukhozi Telecom is fictional, and its systems and figures are illustrative.
What sits under the line
Backup and restore, tested. The gateway's database holds keys, budgets, routes and usage history. Ukhozi backed it up from day one but had never restored it. The first restore test, into a separate environment, revealed that encrypted key material needed a secret that was not in the backup. That was a two-hour fix in a test, and would have been a multi-day outage in a real incident.
Upgrade and rollback path. Open-source gateways in this space release often, because providers change their APIs often. Staying current matters for security and compatibility; upgrading carelessly breaks routes. Ukhozi settled on a monthly upgrade window, with upgrades applied to a staging gateway first, the Part 2 evidence tests rerun automatically, and a documented rollback to the previous version and database schema. Part of the reason to pin versions is supply-chain risk, which the next series, The AI Software Supply Chain, takes up directly.
Capacity and rate planning. Two kinds of capacity matter: the gateway's own (CPU, memory, database connections, especially with streaming), and the provider quota behind it, which is shared by every application. When the call-centre summarisation volume doubled, it consumed rate limit the dispute agent needed. Ukhozi now reviews quota headroom monthly and reserves a share for customer-facing routes.
Incident response runbook. Who is paged when the gateway is down or slow? How do they tell a gateway fault from a provider fault? How do they bypass the gateway in an emergency, and who may authorise that? Ukhozi's runbook answers all three, including a tested emergency route for the dispute agent that goes directly to the provider with a temporary key and is logged.
Quarterly access reviews. Who can create keys, change routes or read stored content? Admin access to a model gateway is powerful: it can redirect traffic or raise a budget. Ukhozi reviews admin access quarterly alongside its other privileged access, and requires two people for route changes on customer-facing applications.
Cost reconciled to invoices. The gateway's cost figures are estimates based on its own token counting and price tables. The provider's invoice is the truth. Ukhozi reconciles the two monthly and investigates any gap above a set tolerance, which is how it found the tokeniser mismatch in Part 2 and, later, a price change the gateway's table had not picked up.
A minimum operating rhythm
Operations become reliable when they are scheduled. Ukhozi's minimum rhythm, once three applications depended on the gateway:
The rhythm takes perhaps a day a week of one engineer's time at Ukhozi's scale (illustrative). That is the real cost of the gateway, and it is worth stating up front: in licence terms the gateway was free, and in operating terms it was a small but permanent commitment.
Ownership, not heroics
The most common failure is not a technical one. The gateway was set up by one capable engineer, who knows how it works, and nobody else does. When that engineer is on leave, upgrades stop and incidents wait. One hero is not a delivery model makes the same point about forward-deployed teams.
Ukhozi wrote down four ownership facts for the gateway:
| Question | Answer |
|---|---|
| Service owner | Platform team lead |
| On-call | Platform rota, two engineers trained per shift |
| Budget for running it | Platform team cost centre, reviewed annually |
| Who decides to retire or replace it | Architecture board, on the platform lead's recommendation |
The last row matters more than it seems. A gateway that has become critical infrastructure needs a decision path for replacing it, for example with a managed service, without that decision being forced by an incident.
Self-host, managed or cloud-native
The operating load is the main input to a build-or-buy decision. A managed gateway service or a cloud provider's API management with model policies shifts upgrades, availability and some security work to the vendor, in exchange for cost, less control and new data-processing terms. Self-hosting gives control and flexibility and keeps all of the operating work. Neither is right in general; the honest comparison includes the operating rhythm above, not just the licence line.
The short version
Installation is not operation. A demo is a demo until someone runs it.
Test the restore. A backup you have never restored is a hope.
Upgrade on a schedule, with rollback. Fast-moving software punishes both neglect and haste.
Reconcile to invoices. The provider's bill is the truth.
Name the owner. On-call, budget and the retirement decision.
Previous: Part 6, do your observability tools store the prompt? Next: Part 8, do not add a gateway without a requirement. All parts are on the series page.
How CloudNala can help
We help platform teams turn a working gateway into an operated service: restore tests, upgrade and rollback procedures, capacity and quota planning, incident runbooks, access reviews and monthly cost reconciliation, with owners named for each. Where the operating load is the deciding factor, we help compare self-hosted, managed and cloud-native options on the full cost.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za