AI gateway, API gateway and agent gateway: three different jobs

By the end of its first year with AI in production, Ukhozi had three things called "gateway" in its architecture diagrams. The API gateway in front of its customer app had been there for years. The model gateway was the subject of this series. And a newer tool gateway now sat between the billing-dispute agent and the systems it could act on. In an architecture review, someone reasonably asked why one product could not do all three, since two of the vendors on the shortlist said theirs could.

The question is fair, and sometimes the answer is that one product can host two of the jobs. But the jobs themselves do not merge. Each gateway governs a different kind of traffic, applies policy to a different unit and protects against a different risk. A product that does all three is still three sets of policy to design, test and operate.

Ukhozi Telecom is fictional, and its systems and figures are illustrative.

Three gateways side by side

Some products do two of these jobs. The jobs remain different, and a product that does all three is still three sets of policy to design and test.

The API gateway sits in front of your own services and governs calls into them. Its unit of policy is the route and the consumer: which client may call which endpoint, at what rate, with which authentication. Its main risk is exposed backends, so it does authentication, throttling, request validation and routing to the right service version. Architecture in Plain English explains it in full.

The model gateway sits between your applications and model providers and governs model calls. Its unit of policy is the model, the tokens and the key: which application may call which model, how many tokens, within which budget, with what redaction. Its main risk is data leaving for providers and cost running away. That is the subject of Parts 1 to 8.

The agent or tool gateway sits between agents and the tools they act through, often as an MCP gateway. Its unit of policy is the operation and its scope: which agent may discover and invoke which operation, with which delegated identity, parameters and approvals. Its main risk is side effects, such as a credit issued, a message sent or a record changed. The enterprise MCP gateway and the gateway is a control point, not a trust boundary cover it.

The differences show up most clearly in the limits each applies. An API gateway counts requests per second. A model gateway counts tokens and rand. A tool gateway counts actions and asks whether this one needs a person's approval.

One request through all three

A Ukhozi customer asks the app to explain and dispute a bill line. Each gateway makes a different decision about the same request.

A Ukhozi customer opens the app and asks the dispute agent to explain a R349 charge and dispute it.

The API gateway checks that the app's session is valid and within rate limits, and routes the request to the dispute service. It neither knows nor cares that AI is involved.
The dispute agent needs a model to understand the question. Its call goes through the model gateway, which checks the agent's key may use the chosen model, redacts the customer's ID number from the prompt, applies the budget and records tokens and cost.
The agent decides to open a dispute. That call goes through the tool gateway, which checks that this agent may invoke "open dispute" for this customer's account, that the amount is within the limit for automatic disputes, and that no approval is needed.
The billing system still authorises the dispute itself, as every downstream system must.

Each decision is different, made by a different owner, and recorded in a different log. If Ukhozi merged them into one product, it would still need all three policies, and it would need to make sure a misconfiguration in one did not weaken the others.

When to combine, and when not to

Combining jobs in one product can be sensible:

An API management platform with model-specific policies (token limits, content safety hooks, semantic caching) can serve as both API and model gateway, especially where the organisation already runs it well.
Some model gateways now also proxy MCP traffic, which can suit a small number of agents.

Reasons to keep them apart:

Different owners. The API gateway usually belongs to the integration team, the model gateway to the AI platform team, and tool policies to system owners and risk. One product with three owners needs very clear configuration boundaries.
Different blast radius. A failure in the tool gateway should not take down customer API traffic.
Different change rates. Model routing changes weekly; API routes change with releases; tool permissions change with business rules.

Whatever the product choice, write the three policies separately, test them separately, and make sure each downstream system still defends itself.

What the series adds up to

Across nine parts, the argument has been consistent:

The model gateway is one layer of nine. Draw the whole platform, and map the bypass paths.
Prove it small. One consumer, one provider, one model, seven pieces of evidence.
Keys are for accountability. Not identity.
One layer owns the retry. With nested timeouts and idempotent actions.
Failover is per route. Evaluated, equivalent terms, identical contract, or fail closed.
Logging is data handling. Decide per field, apply everywhere, prove deletion.
Operate it. Restore tests, upgrades, capacity, incidents, access, reconciliation.
Require a requirement. A thin client is a legitimate answer.
Three gateways, three jobs. Combine products if you like, never policies.

A model gateway, well run, is one of the most useful pieces of AI infrastructure an organisation can own. It is not the platform, and it is not governance by itself.

Previous: Part 8, do not add a gateway without a requirement. Back to Part 1, the model gateway is not the AI platform. All parts are on the series page.

How CloudNala can help

We help architecture and platform teams separate the gateway jobs in their own estate: which traffic each gateway governs, which policies belong where, which owners are responsible, and whether combining products is sensible. The result is an architecture where API, model and tool controls each do their own job, and no single misconfiguration removes them all.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za