The advisory usually arrives sideways. Someone posts a link in a team channel, or a dependency scanner raises an alert, or a provider emails about unusual key usage. It names a package, one or two versions and a time window. The question in everyone's head is "are we affected?", and the instinct is to upgrade immediately and move on.
That instinct is understandable and wrong in the wrong order. Upgrading first replaces the evidence you need to answer the question. Rebuilding before rotating secrets produces a clean service that reads stolen keys. The useful response is a sequence, and the sequence is worth agreeing before the day it is needed.
Langeberg Software is fictional, and its systems and figures are illustrative. In this scenario Langeberg reacts to a fictional advisory modelled on the public March 2026 incident described in Part 1: a malicious release of the open-source proxy package behind its model gateway was live on the public registry for about 45 minutes, and the payload harvested credentials from the environment it ran in.
The sequence
1. Identify exposure
Find every place the named versions could have been installed. Not just production: CI runners, scheduled jobs, evaluation pipelines, developer machines, notebooks and container builds. Use evidence, in this order of reliability: lockfiles and image digests in your repositories and registry; build and CI logs for the window; package cache and proxy logs if you run an internal mirror.
At Langeberg this took 40 minutes. The production gateway was pinned by digest to an older release and was clear. The nightly evaluation job had run before the window. One developer's evaluation run, started in CI at 12:50 South African time, had installed the proxy package unpinned and resolved the bad version. Its container failed to start because of an unrelated dependency conflict, which created a question Part 4 deals with in full: does a failed start mean it was safe?
2. Isolate
Stop affected workloads and cut their network egress before anything else changes. If a runner or host may still be compromised, take it out of the pool. Isolation buys time: whatever the payload does next, it now does it without a route out.
3. Preserve evidence
Copy logs, images, the runner's file system if possible, and registry history to a location the affected credentials cannot reach. CI providers often expire logs in weeks; Kubernetes deletes a pod's local state when the pod goes. This step is the one most often skipped, and it is the one that later lets you say "exposed" or "not exposed" with confidence instead of guessing.
4. Rotate secrets
Rotate every secret the affected process could have read, not only the ones you think it used. Then confirm the old ones are revoked, not merely replaced.
5. Verify artifacts
Check that what is running everywhere else is what you believe it is: compare image digests and file hashes against your records and against the publisher's list of known-good versions, if one exists.
6. Rebuild
Rebuild affected images and environments from pinned, known-good versions in a clean pipeline. Do not reuse the runner that ran the bad version until it has been rebuilt.
7. Monitor
Watch for use of the old secrets, unusual provider spend, unusual egress from build infrastructure and new advisories from the same campaign. Attackers who harvest credentials often use them days later.
8. Record
Write down the timeline, what was exposed, what was rotated, the decisions taken and the evidence behind each. Then change something: a pin, a retention setting, a guard in the pipeline. An incident that changes nothing will recur.
Why the order matters
Three ordering rules carry most of the value.
Preserve before rebuild. A rebuild overwrites the hosts you need to examine. If you rebuild first, the question "did the payload run?" becomes unanswerable.
Rotate before rebuild. A freshly rebuilt service that reads an already-stolen key is not clean. Rotation is what actually removes the attacker's access.
Isolate before rotate. Rotating secrets on a host that may still be running the payload can hand the new secrets straight to the same exfiltration path.
None of this needs a large security team. It needs the steps written down, an owner for each, and one rehearsal.
How far does the rotation reach?
The hardest judgement in the first hour is scope. Rotate too little and the attacker keeps access. Rotate everything in the company and you cause an outage of your own making.
Langeberg worked through the questions for the one affected job. The bad version was installed. Whether a Python process had started with it present was unknown, because the container crashed early and the logs did not say. Unknown was treated as yes. The job's environment could read:
- the staging gateway's master key, which could call both model providers;
- a read-only credential for a test database holding synthetic invoices;
- the CI runner's cloud role, which could read a storage bucket of test documents;
- a token for the internal package registry.
All four were rotated within two hours. The staging gateway key was the most important, because it carried real spending authority on paid provider accounts, and provider usage for that key was checked for the full window and the following week. The package registry token mattered more than it first appeared: a stolen publishing token is how a single compromise turns into the next one.
The principle Langeberg adopted is the one in the diagram: when a question cannot be answered from evidence, answer it as yes. An extra rotation costs hours. A missed one costs an unknown amount for an unknown time.
What a runbook needs before the day
The sequence is generic. The runbook has to be specific to your environment, and it is far easier to write on a quiet afternoon than during an incident. At minimum it should hold:
- An inventory of where AI components run, including CI and developer tooling, with how each is pinned. Part 6 covers what that inventory should contain.
- A map from each environment to the secrets it can read. This is what turns "rotate everything" into a scoped list.
- Log retention chosen on purpose for CI runs, egress and runtime, long enough to cover the time between a compromise and its disclosure.
- A kill switch for the gateway, so all AI traffic can be stopped with one change if the gateway itself is in question. Part 7 covers it.
- Named owners and a channel, so that the first hour is spent acting rather than finding people.
Rehearse it once with a fictional advisory. Langeberg's rehearsal found that two of the four secrets in its scenario had no documented rotation procedure, which is a much better time to find out.
Where this connects
AI-assisted triage can help during an incident, with the boundaries discussed in AI-assisted incident response. The audit trail that makes this sequence possible is the same one argued for in Your agent spent the money. Can you prove why?: records made at the time, not reconstructed afterwards.
Next: Part 4, The container will not start. Can you still prove it was safe?. All parts are on the series page.
How CloudNala can help
We help teams write and rehearse a containment runbook for their AI stack: the exposure inventory, the environment-to-secrets map, retention settings and a tabletop exercise against a realistic advisory. The deliverable is a runbook your own engineers have already used once, plus the short list of gaps the rehearsal exposed.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Supply Chain Review or write to us at consult@cloudnala.co.za