Somewhere in most AI programmes there is a slide with a box labelled "AI gateway" in the middle and every application drawn neatly on one side, every model on the other. The message is reassuring. All AI traffic flows through one place, so that place can enforce policy, control cost, log everything and switch providers when needed. Buy or deploy the gateway, and the platform is done.
The gateway is genuinely useful. It is also one layer of a larger system, and the slide hides the rest of that system. Agents still need a runtime. Tools still need authorisation. Memory and retrieval still need data owners. Quality still needs evaluation. And a surprising amount of the organisation's AI never touches the gateway at all.
CloudNala's view is simple: a model gateway is a control point for model traffic, not a governance model for AI. Treat it as the first, rather than the last, piece of platform architecture, and be explicit about what it does not see.
This series takes the gateway apart in nine parts: what it is for, how to prove it works, keys, retries, failover, logging, operations, when not to build one, and how it differs from API and agent gateways. One fictional South African operator carries the examples.
Ukhozi Telecom is fictional, and its systems and figures are illustrative. Ukhozi is a mid-size mobile and fibre operator. Eighteen months after its first AI pilot, it runs nine AI applications: a store-support assistant for retail staff, a billing-dispute agent in the customer app, call-centre summarisation, a network-fault summariser for the operations centre, sales-quote drafting, an HR policy assistant, marketing copy inside a SaaS marketing suite, a coding assistant for developers and a contract-review helper in legal. A four-person platform team runs a self-hosted open-source model gateway in front of two model providers.
When the chief information officer asked "is all our AI governed?", the platform team's first answer was "yes, it goes through the gateway". The review that followed is the backbone of this series.
What a model gateway actually does
A model gateway sits between applications and model providers. In its simplest form it is a reverse proxy that speaks the providers' APIs. In practice the useful features cluster into six jobs:
- Access. Applications get gateway credentials instead of provider keys, so provider secrets live in one place.
- Routing. A logical model name maps to a real provider and model, which makes switching or splitting traffic a configuration change.
- Limits. Rate limits, token limits and spend budgets per application, team or key.
- Policy hooks. Redaction, content filtering or blocking of certain requests before they leave the organisation.
- Telemetry. Who called which model, how many tokens, how long it took, what it cost, and whether it failed.
- Resilience. Timeouts, retries and fallback to a second model or provider.
Each of those is worth having. None of them, on its own, makes an AI system safe, correct or valuable. The gateway sees a request go out and a response come back. It does not see what the agent did next.
The platform has nine layers, and the gateway is one
The clearest way to correct the slide is to draw the whole platform and mark which layer the gateway occupies.
Reading the map from the top:
Applications and channels own the user experience and the business outcome. A gateway cannot tell whether the billing-dispute agent resolved the customer's problem.
The agent runtime decides what to do next: which tool to call, when to stop, when to ask a person. That loop runs inside the application or an orchestration service, not inside the gateway. A gateway sees ten model calls; it does not see that the agent was stuck in a loop for nine of them.
Tools, MCP servers and APIs are where actions happen. Opening a dispute, issuing a credit or changing a contract runs through the billing system's API, and that API must authorise the action itself. We covered why in the gateway is a control point, not a trust boundary, and for MCP specifically in the enterprise MCP gateway.
Memory and retrieval decide what context the model sees. If a retrieval index ignores document permissions, every model call carries data the user should not see, and the gateway passes it through faithfully. See permission-aware retrieval for the design.
Policy and guardrails span several layers. The gateway can host some of them, such as redaction of ID numbers before a prompt leaves. Others, like "the agent may not promise a refund above R500", belong in the agent or the tool.
Evaluation tells you whether outputs are good enough. A gateway can record outputs for later evaluation; it cannot decide what "good" means for a dispute summary.
Observability and cost draws on the gateway's telemetry, but also on agent traces, tool logs and business events. A gateway's cost figure per application is necessary and not sufficient.
Model providers sit under contract. The contract, not the gateway, governs what the provider may do with the data.
The bypass paths
The second correction is about coverage. A gateway governs only the traffic that passes through it, and at most organisations a large share of AI use does not.
Ukhozi found all three bypass routes in its first week of looking.
Direct SDK calls. The network-fault summariser had been built by an operations engineer before the gateway existed. It called a provider directly with a key stored in the application's configuration. It worked, so nobody had migrated it. Its usage did not appear in any gateway dashboard, and its key had never been rotated.
Embedded vendor AI. The marketing suite and the contract-review helper are SaaS products with AI features built in. Their prompts go from the vendor's servers to the vendor's chosen model provider. The gateway will never see them. Governing them is a vendor-management and contract exercise: data-processing terms, regions, retention and the ability to switch the feature off.
Shadow AI. Staff use consumer AI services on personal subscriptions. A gateway does nothing here. Acceptable-use policy, approved alternatives that are genuinely good enough, and data-loss controls on the endpoint are the tools that work.
None of this makes the gateway less useful. It makes the claim "all our AI goes through the gateway" false, and a false claim in a board pack is worse than an honest gap. The Agent Evidence Layer covers the evidence side of the same problem: Weaverbird Agri's gateway totals never matched its invoices because one connector skipped the gateway.
What Ukhozi changed
The review produced a one-page coverage statement instead of a yes. For each of the nine applications it recorded:
| Application | Through gateway? | Agent or tool controls | Other governance |
|---|---|---|---|
| Store-support assistant | Yes | Read-only tools | Evaluation set, owner |
| Billing-dispute agent | Yes | Billing API authorises; approval above a limit | Evaluation set, owner |
| Call-centre summaries | Yes | None needed | Retention rule for transcripts |
| Network-fault summariser | No, migrating | Read-only | Key rotated, owner named |
| Sales-quote drafting | Yes | Draft only | Pricing check by a person |
| HR policy assistant | Yes | Read-only, permission-aware retrieval | HR owner |
| Marketing suite AI | No, vendor-embedded | n/a | Contract and settings review |
| Coding assistant | No, vendor service | n/a | Enterprise plan, repository scope |
| Contract-review helper | No, vendor-embedded | n/a | Legal owner, data terms |
The statement told the chief information officer what was actually true: five applications governed at the model layer, four governed by other means, one migration in flight, and specific owners for each. It is a less comfortable answer than "yes". It is one the organisation can defend.
Where the gateway earns its place
None of this argues against gateways. Ukhozi's gateway paid for itself quickly in three ways:
- Provider keys left the applications. Five applications holding provider keys became one gateway holding them, with scoped gateway keys for each application.
- Cost became attributable. Spend per application and per team was visible weekly instead of as one provider invoice a month later.
- A provider switch became configuration. When the team evaluated a cheaper model for call-centre summaries, it changed a route and compared results, without touching the application.
Those are real benefits. They are benefits of a well-run piece of infrastructure, not of a governance programme. The rest of the series is about getting those benefits without overstating them. Part 9 returns to how the model gateway differs from the API gateway explained in Architecture in Plain English.
Questions to ask about your own gateway
- Which AI applications send traffic through it, and which do not? Do you have the list in writing?
- For each application that does, which controls live in the gateway, and which live in the agent, the tools and the data layer?
- For each that does not, what governs it instead?
- Who owns the gateway as a running service, with on-call, upgrades and a budget?
- If the gateway disappeared tomorrow, which controls would vanish with it, and which would remain?
The last question is the most revealing. If the honest answer is "nearly all of them", the gateway has become a single point of governance failure, and the platform has a gap below it.
The short version
The gateway governs model traffic. It does not govern agent behaviour, tool actions, retrieval permissions or quality.
Draw all nine layers. Give each one an owner before claiming the platform is governed.
Map the bypass paths. Direct SDKs, embedded vendor AI and shadow AI each need their own control.
Replace "yes" with a coverage statement. It is the answer a board can rely on.
Next: Part 2, one model, one provider, one consumer: the first useful proof. All parts are on the series page.
How CloudNala can help
We help platform and architecture teams map their AI platform layer by layer: which applications use the gateway, which bypass it, which controls belong in the gateway and which belong in agents, tools and data, and who owns each layer. The output is a coverage statement and a short list of gaps that your security, risk and finance teams can act on, rather than a diagram that implies more control than exists.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za