The Model Gateway Is Not the AI Platform
A model gateway can centralise model access, keys, budgets, routing and telemetry. It does not govern agent runtimes, tools, memory, data access, evaluation or the AI that never passes through it. This nine-part series maps the gateway's place in the wider AI platform and the bypass paths around it, sets out a narrow first proof with recorded evidence, and covers virtual keys, retry ownership, per-route failover, prompt logging as a data-handling decision, the work of operating a gateway, when not to build one, and how model, API and agent gateways differ. One fictional South African telecoms operator, Ukhozi Telecom, carries the examples, and all figures are illustrative.
9 articles
- Part 1The model gateway is not the AI platformA model gateway can centralise model access, keys, budgets, routing and telemetry. It does not govern agent runtimes, tools, memory, data access, evaluation or the AI that never passes through it. Part 1 maps what the gateway controls, what sits outside it, and the bypass paths that make a gateway-only strategy look safer than it is.8 min read
- Part 2One model, one provider, one consumer: the first useful proofMost gateway rollouts try to prove everything at once and end up proving nothing. Part 2 sets out a deliberately narrow first proof: one consumer, one provider, one model, and seven pieces of recorded evidence covering authentication, streaming, attribution, failure tracing, throttling, timeouts and redaction.6 min read
- Part 3Virtual keys are an accountability mechanismGateway keys scoped to an application, with model allowlists, budgets, rate limits and expiry, are one of the most useful things a model gateway offers. They are also easy to mistake for an identity model. Part 3 explains what a virtual key can answer, what it cannot, and the lifecycle that keeps keys accountable.5 min read
- Part 4Who owns the retry?The application, the agent framework, the gateway and the provider SDK can each retry a failed model call. Left uncoordinated, three sensible settings multiply into dozens of calls, overload a degraded provider, inflate the bill and can repeat an action with side effects. Part 4 explains retry storms, nested timeouts and why one layer must own the retry.6 min read
- Part 5Failover can improve availability and damage consistencyRouting to a second model when the first is unavailable keeps requests flowing. It can also change refusals, break structured output, shrink the context window, move data to a different region and alter cost. Part 5 explains when failover is safe, when a route should fail closed instead, and how to decide per route.5 min read
- Part 6Do your observability tools store the prompt?Gateway logs, tracing tools, provider monitoring, security exports and backups can each keep a copy of every prompt and response, including personal information users typed in. Part 6 explains why logging is a data-handling decision, how to decide per field what to store, redact or never keep, and how to prove deletion works.5 min read
- Part 7A working admin screen is not an operated platformA gateway demo shows keys, models and a dashboard. Running one in production means backup and restore, upgrades and rollback, capacity planning, incident response, access reviews and reconciling costs to invoices, with named owners. Part 7 sets out what operating a model gateway actually takes, and a minimum operating rhythm.5 min read
- Part 8Do not add a gateway without a requirementA model gateway is a shared runtime on the critical path. For one application calling one provider, a thin internal client often does the job with nothing new to operate. Part 8 sets out five requirements that justify a gateway (routing, portability, attribution, policy and resilience) and how to decide honestly when none of them is true yet.5 min read
- Part 9AI gateway, API gateway and agent gateway: three different jobsVendors increasingly sell one product as an API gateway, an AI gateway and an agent or MCP gateway. The jobs remain different: each governs a different kind of traffic, applies policy to a different unit and protects against a different risk. Part 9 separates the three, follows one customer request through all of them, and closes the series.5 min read
Working through any of this in your own organisation?
These articles come out of delivery work rather than theory. If the problems in this series look familiar, a short scoping conversation is usually the quickest way to work out what is worth doing first.
Talk to CloudNala