Not every organisation that reads the first seven parts of this series should build a model gateway. Some should, urgently. Others are adding a shared component to the critical path of a single application, because a reference architecture had a box for it.
Ukhozi did need one, eventually. When it started, with two AI applications both calling one provider, it did not. A sister company in the same group, a fictional 120-person logistics software business with one AI feature in its product, went through the same question a year later and made the opposite decision, correctly.
CloudNala's position: a gateway is justified by a requirement you can name today, not by the architecture you expect to have in two years. When none of the requirements holds, a thin internal client is the better choice, and the decision should be revisited when one appears.
Ukhozi Telecom and its sister company are fictional, and their systems and figures are illustrative.
Five requirements that justify a gateway
Routing. More than one model or provider is genuinely used, per use case or across them, and changing the mapping should not require application releases. Ukhozi reached this when it moved call-centre summaries to a cheaper model while keeping the dispute agent on the stronger one.
Portability. There is a real plan, with a date or a trigger, to switch providers. "We might want to switch one day" is not a plan. A procurement review next year that could change the provider is.
Attribution. Several teams share one provider account and one bill, and finance needs that bill split reliably by application or cost centre. With one application, the provider's own reporting is enough.
Policy. A control must apply to every AI consumer consistently, for example redaction of ID numbers before any prompt leaves the organisation. With one consumer, that control can live in its code. With nine, central enforcement is far more reliable than nine implementations.
Resilience. There is an outage scenario that a fallback would actually cover, and routes that can safely fail over (see Part 5). If none of your routes can fail over safely, the resilience argument does not apply yet.
A practical rule: when two or more of these hold, a gateway earns its place. When one holds, look at whether a simpler mechanism covers it. When none holds, do not build one.
What a thin client gives you
The alternative is not "every developer calls the provider however they like". It is a small shared library, owned by one team, that every application uses to call the model.
A good thin client handles:
- loading the provider credential from the organisation's secret store, never from code;
- consistent timeouts and a single retry policy (the Part 4 rules apply here too);
- a correlation ID on every call;
- structured logging of metadata, with the content rules from Part 6;
- a tag on every request for the application and cost centre, if the provider supports it.
It adds no new runtime to operate, no new failure point and no new data store. Its weakness is that policy changes need application releases, and it cannot enforce anything across applications that do not use it. Those are exactly the weaknesses that become important when the second and third applications arrive, which is when the requirements above start to hold.
The sister company's single AI feature summarised shipment exceptions for its customers. One model, one provider, one team, one bill, and redaction implemented once in that feature's code. A gateway would have added a service to operate and a hop to every request, and given them nothing they needed. They built the thin client, wrote down the five requirements, and agreed to revisit the decision when any of them changed.
Signs the decision should be revisited
- A second team starts building an AI feature.
- Finance asks for the AI bill split, and the answer takes a spreadsheet.
- A second provider or model enters production.
- Security or privacy asks for a control "on all AI traffic".
- A provider outage causes a customer-facing incident.
Any one of these is a reason to reopen the question, not an automatic reason to build. The thin client also makes the migration easier when the time comes: if every application calls the model through one library, pointing that library at a gateway is a contained change.
When you already have one
Some organisations have a gateway and only one real consumer, because it was installed early. The honest options are to keep it, accepting the operating load from Part 7 in return for being ready; to pause it and use a thin client until requirements appear; or to replace it with a managed option whose operating load is lower. Any of these can be right. Keeping a gateway nobody operates properly is the one option that is not.
The short version
Name the requirement. Routing, portability, attribution, policy or resilience.
Two or more, build it. One, look for a simpler mechanism. None, do not.
A thin client is a real architecture. Credentials, timeouts, retries, correlation and logging in one library.
Write down the triggers. Revisit the decision when one fires.
Previous: Part 7, a working admin screen is not an operated platform. Next: Part 9, AI gateway, API gateway and agent gateway: three different jobs. All parts are on the series page.
How CloudNala can help
We help architecture teams make the gateway decision on evidence: which of the five requirements hold today, whether a thin client covers them, what a gateway would cost to operate, and which triggers should reopen the question. The output is a short decision record that holds up when the next reference architecture with a gateway box arrives.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za