One model, one provider, one consumer: the first useful proof

A gateway proof of concept usually starts with ambition. Connect three applications, configure two providers, turn on caching, enable fallback, wire up the dashboards, and show the steering committee one screen with everything flowing. The demonstration works. Six weeks later, the first production incident reveals that streaming responses were being buffered, that one application's usage was being attributed to another, and that nobody had checked what the gateway wrote to its logs.

The problem was not the gateway. It was the proof. When everything is tested at once, a pass tells you very little about any single property, and a failure is hard to trace to a cause.

CloudNala recommends the opposite: prove the smallest complete path first, with evidence for each property, and add one element at a time.

Ukhozi Telecom is fictional, and its systems and figures are illustrative. In Part 1 Ukhozi's platform team mapped nine AI applications and found that five could sensibly sit behind its model gateway. This part covers how it proved the gateway before moving them.

Choose the path

The first proof should have exactly one of each:

Ukhozi's first proof: the store-support assistant, one approved model, one provider. Everything the gateway claims must be demonstrated on this path before a second consumer joins.

One consumer. Ukhozi chose the store-support assistant, used by retail staff to look up plan rules and device policies. It is internal, read-only, moderately busy and streams its answers, which makes it a good test of the hard parts without customer exposure.

One provider and one model. The model the assistant already used, on the provider it already used. Changing the model during the gateway proof would mix two questions: is the gateway working, and is the new model good enough?

Nothing else. No fallback provider, no semantic cache, no agent tool traffic, no second application. Each of these adds a failure mode the team would then have to separate from the gateway's own behaviour.

The narrow scope also gives a clean comparison. Ukhozi could run the assistant through the gateway and directly for a week and compare latency, errors and token counts side by side.

Seven properties, each with evidence

"It works" is not evidence. For each property, the team defined a test it could run and record.

Each item is demonstrated with a recorded test, not asserted from the product documentation. One missing item keeps the gateway at one consumer.

1. The caller is authenticated. A request without a valid gateway key is refused. A request with a key scoped to a different model is refused. A revoked key stops working within the stated time. The evidence is a short test log, not a screenshot of the settings page.

2. Streaming is intact. The assistant streams tokens to the staff member's screen. Some proxies buffer the full response before forwarding it, which turns a two-second first token into a twelve-second wait. Ukhozi measured time to first token directly and through the gateway, under load.

3. Usage is attributed correctly. Tokens and cost recorded by the gateway for the assistant's key match the provider's own usage report for the same period, within a tolerance the finance analyst agreed. This is the test most proofs skip, and the one finance cares about most.

4. A failure can be traced. The team forced a provider error and a malformed request and confirmed it could follow one failed request from the assistant's log, through the gateway's log, to the provider's request ID. Without a shared correlation ID, every incident becomes an argument between teams about whose log is right.

5. Throttling works as configured. A load test above the key's rate limit produced the gateway's limit response, the assistant handled it gracefully, and the gateway did not pass the excess to the provider. The team also checked what happens when the provider throttles first, which is a different response from a different place.

6. Timeouts are honoured. A slow provider response hit the configured timeout and returned a clear error, rather than holding the staff member's screen open for minutes. Part 4 covers why the timeout values matter as much as their existence.

7. Redaction is proven. The gateway was configured to mask South African ID numbers and phone numbers in prompts. The team sent test prompts containing both, then inspected what the provider received (through the provider's own logging where available) and what the gateway stored. Redaction that is configured but never tested is a belief, not a control.

What the proof found

Three of the seven properties failed on the first run, which is normal and is the point of the exercise.

Streaming was buffered because a load balancer in front of the gateway had response buffering enabled. Attribution was off by about 6% (illustrative) because the gateway counted tokens with a different tokeniser from the provider's billing. Redaction caught ID numbers written as thirteen continuous digits but not ID numbers written with spaces, which is how many staff typed them.

Each failure was cheap to fix because it was isolated. Had the proof involved three applications and two providers, the buffering would have looked like a provider latency problem, and the attribution gap would have been blamed on whichever application was busiest.

The gate to the second consumer

Ukhozi's rule was that all seven items must be recorded and reviewed before a second application joins. The review is short: the platform lead, a security engineer and the finance analyst look at the evidence and sign it off. If one item is missing, the gateway stays at one consumer.

After that, elements are added one at a time, each with its own small proof:

A second consumer (the HR policy assistant), proving that attribution and limits separate cleanly between two keys.
A fallback provider, which, as Part 5 shows the hard way, should be enabled route by route once each route's consistency questions are answered.
Agent traffic from the billing-dispute agent, which makes many calls per case and stresses budgets and timeouts differently.
Caching, last, because it changes both cost and data handling.

The whole sequence took Ukhozi about ten weeks. A big-bang proof would have looked faster in week two and slower by week twelve.

A checklist you can reuse

The short version

Small proofs produce clear answers. One consumer, one provider, one model.

Evidence, not assertions. Every property gets a recorded test.

Expect failures. Finding three on the first run is the proof working.

Add one element at a time. Each addition gets its own small proof.

Previous: Part 1, the model gateway is not the AI platform. Next: Part 3, virtual keys are an accountability mechanism. All parts are on the series page.

How CloudNala can help

We design and run gateway proofs that produce evidence rather than demonstrations: scoping the first path, writing the tests for authentication, streaming, attribution, tracing, throttling, timeouts and redaction, and running the review that decides whether the next consumer can join. The result is a gateway your security and finance teams have seen working, not just configured.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za