There is a particular kind of relief that follows a failed deployment during a security incident. The affected job installed the bad version, but the container crashed on start. It never served a request, never called a provider, never touched production. Someone says "we got lucky" and the ticket moves towards closed.
Sometimes that is true. The problem is that the statement is usually made without evidence, and the mechanisms that make modern supply-chain payloads effective are specifically the ones that run before your application does.
Langeberg Software is fictional, and its systems and figures are illustrative. In Part 3, one of Langeberg's CI evaluation jobs installed a malicious release of its gateway's proxy package during a 45-minute window. The container then failed to start because of an unrelated dependency conflict. This part is about how the team decided what that crash did and did not prove.
Why a crash proves less than it seems
Most package ecosystems have ways to run code before, or regardless of, the application's own entry point.
Install-time hooks. Some packaging formats can run code during installation. In those cases the payload runs in the build or CI environment at install, before any container is even assembled.
Interpreter start-up hooks. Python processes .pth files in its site-packages directory every time the interpreter starts. A .pth line that begins with import is executed. That means code can run when any Python process starts in that environment, including the one that then crashes on a missing dependency. The public March 2026 advisory for LiteLLM specifically named a .pth file in one of the malicious versions for this reason, as described in the project's security update.
Import-time code. Any module the application imports before failing gets its top-level code executed. A crash halfway through start-up may come after the interesting part.
Exfiltration needs no traffic. A credential stealer does not need the service to receive a request. It needs a few seconds of process time and an outbound connection.
So "the container crashed" moves the question rather than answering it. The real question is: did a process start with the bad package present, and could it reach the network?
The evidence that can answer it
Each source answers a different part of the question, and each has its own way of disappearing.
Registry history and lockfiles show which version resolved. They answer "was it installed?" and they are usually kept indefinitely. At Langeberg, the CI job's install log showed the bad version resolving at 12:51.
Image digests and SBOMs show what was inside an image, if the image was built and stored. Ephemeral CI containers are often never pushed anywhere, so there may be no image to inspect. If you sign and store SBOMs for every build, this becomes a lookup.
CI run logs show when the job started, which steps ran and how it failed. Many CI services expire these after a few weeks by default. Langeberg's logs survived only because the incident was caught within days.
Egress and DNS logs are the closest thing to a direct answer. If the runner's network made a connection to an unexpected host during the window, the payload almost certainly ran. If flow and DNS logs show no outbound connections at all from that runner during the job, that is strong evidence it did not get out. Langeberg had flow logging on the CI subnet only because someone had switched it on months earlier to investigate a cost spike.
Runtime and audit logs show what the process did, but only if they were shipped somewhere before the pod or runner was deleted. Local logs on an ephemeral runner are gone when the runner is recycled.
What Langeberg found
Working through the sources took most of a day.
- The install log confirmed the bad version was installed at 12:51.
- The job log showed the Python interpreter starting, importing the test harness and failing on a version conflict in a different library about four seconds later. So a Python process did start with the package present.
- The CI subnet's flow logs showed one outbound HTTPS connection from that runner during those seconds to a host that was neither a model provider nor the package registry.
That settled it. The payload had very probably run and connected out. The "we got lucky" view would have left the four secrets from Part 3 in place. Because the team had already treated unknown as yes and rotated within two hours, the evidence confirmed a decision rather than forcing a late one, and provider usage logs for the staging key showed no use from unfamiliar sources before rotation.
Had the flow logs not existed, the honest conclusion would have been "possibly exposed, scope unknown", with the same rotation and a longer monitoring period. That is still a defensible conclusion, as long as it is written down as uncertainty rather than dressed up as safety.
Make the evidence exist before you need it
The general lesson is about retention and placement, decided in advance.
| Evidence | Default risk | What to set |
|---|---|---|
| CI logs | Expire after weeks | Retain for at least the time between a compromise and typical disclosure, which can be weeks |
| Build images and SBOMs | Ephemeral images never stored | Generate an SBOM for every build, sign and store it even when the image is discarded |
| Egress and DNS logs | Off by default in many clouds | Turn on for build infrastructure and the gateway's network; they are cheap relative to what they answer |
| Runtime logs | Lost with the pod or runner | Ship to central storage the workload's own credentials cannot delete |
| Registry history | Usually kept | Run an internal mirror or proxy so you have your own record of what was fetched, and when |
None of these is AI-specific. What makes them urgent for AI infrastructure is reach: the components in question hold provider keys and see sensitive data, so "possibly exposed" has a larger blast radius.
The wording matters
When reporting the outcome, distinguish three statements:
- "Not exposed: we can show the bad version never entered this environment." This is the strongest claim, and pinning from Part 2 is what makes it possible.
- "Installed, but evidence shows no execution or no outbound connection." Defensible, if the evidence is named.
- "Installed, outcome unknown." Honest, and it should drive the same rotation as a confirmed exposure.
The statement to avoid is "it crashed, so it is fine". That is a conclusion about luck, not about evidence. The Agent Evidence Layer series makes the same argument about agents: an answer that looks complete is not evidence that the work underneath it succeeded.
Next: Part 5, Agent skills and MCP servers are executable dependencies. All parts are on the series page.
How CloudNala can help
We review whether your build and AI infrastructure would leave enough evidence to answer "were we exposed?": log retention, egress logging, SBOM generation and where runtime logs end up. The output is a short set of configuration changes, ranked by how much uncertainty each one removes in a real incident.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Supply Chain Review or write to us at consult@cloudnala.co.za