Container Infrastructure Security: Proving Image Admission and Withdrawing a Release Safely

Consider a hypothetical payment platform releasing checkout:2026.10. The registry has a signature, Kubernetes runs a validating webhook, and the release record says “image verified.” A day later, incident response learns that the release was built from the wrong revision. Nobody can give one reliable answer to where it is running: GitOps shows a tag, some Pods predate enforcement, ARM nodes selected a different platform manifest, and an exception for a system namespace has no owner or expiry.

The scenario is fictional, but the control problem is real. Container infrastructure security in this release context means being able to prove a chain from approved immutable content to the instances actually running, then block matching future admissions, account for restarts inside existing Pods, and replace the instances without unapproved downtime. A signature, an admission policy, and a Pod list are all necessary. None answers the whole question alone.

A Release Evidence Chain, Rather Than One Green Check

A useful assessment does not begin with “is signature verification enabled?” It tests five connected assertions. The release owner approved a specific digest. The signature over that digest has the expected signer identity and issuer. The attestation meets a substantive release policy. Every relevant API request reaches an enforcing admission control. The runtime inventory shows that the approved digests are what run now.

This is narrower than a general Kubernetes security audit, which covers RBAC, secrets, network controls, and workload configuration. It is also narrower than a broad software supply chain security assessment: source and builder properties matter here when they participate in the admission decision for a particular image.

Each assertion needs a separate artifact: a release record containing the digest, signature and attestation verification output, a policy version, an admission event, a snapshot of actual images, and any exception or withdrawal decision. “Cosign passed” does not say which identity was authorized, which attestation fields were evaluated, or whether that policy governed the target cluster.

1. ApproveThe release owner records digests, platforms, expected signer, builder and permitted build parameters.
2. VerifyThe verifier binds signatures and attestations to the digest and evaluates them against a versioned policy.
3. AdmitThe cluster blocks noncompliant requests across the agreed resources, API versions and namespaces.
4. ReconcileInventory compares desired references, admitted digests and the image identities reported for running containers.
5. WithdrawThe team blocks matching new Pods, accounts for container restarts inside existing Pods, and replaces or isolates instances under an evidenced plan.

Digest, Tag and the Platform-Specific Image

A tag is a convenient name, but a registry operator can move it to different content. A digest is a content identifier: a consumer recalculates the hash over retrieved bytes. When an image reference contains both a tag and a digest, Kubernetes uses the digest for pulling. A release record and admission decision should therefore be anchored in the digest, while the tag can remain as a human-readable label.

One string is not enough for a multi-platform image. An OCI image index is a higher-level object containing descriptors for platform-specific manifests. One index digest can select one manifest for linux/amd64 and another for linux/arm64; the platform manifest then identifies configuration and layers. An index signature and a separate child-manifest signature concern different objects. A verified chain of digests binds the signed index to its child content; a separate signature on every child is needed only if the chosen policy requires it.

The acceptance package should retain the index digest, the allowed platform set, and the expected child digest for every platform. A test environment needs nodes, or reliable emulation, for each allowed platform. A successful amd64 test does not prove what an arm64 runtime will select. Post-start inventory should expose both the reference from the PodSpec and the identity resolved by the container runtime; a list of tags alone is insufficient for withdrawal analysis.

What a Signature Proves and What Policy Must Decide

A cryptographically valid signature shows that a particular key or certified identity signed specific content. It does not decide whether that identity deserves authorization in your environment. With keyless signing, at least the expected certificate subject and expected OIDC issuer need to match. A permissive regular expression for either field can reduce strong cryptography to weak authorization. With managed keys, the equivalent question is which key may sign which repository and release channel, who may alter that mapping, and how rotation is governed.

Admission policy must say more than signature valid. For a particular product, useful fields include:

• allowed registry and repository path, plus the approved digest or a controlled way to obtain it from the release record;
• signer identity and issuer, trust root, and transparency-log requirement when the chosen signing scheme uses one;
• required attestation predicate type, its binding to the image subject, and the predicate fields to evaluate;
• canonical source repository, trusted builder.id, buildType, and permitted externalParameters;
• allowed platforms, environments, and Kubernetes resource kinds;
• denied digests, exception owner and expiry, and the conditions for withdrawal;
• behavior when no image policy matches and when the verifier is unavailable.

SLSA gives a useful sequence for provenance verification: validate the envelope signature, ensure the statement subject matches the artifact digest, trust the builder identity, then compare the source, buildType, and external parameters with expectations. Provenance is not a code security analysis and does not promise complete dependency coverage. Authentic provenance can accurately describe a build of an unwanted revision when the policy accepts that source or parameter. Accepting an attestation without inspecting its predicate proves that a document exists; it does not prove that the release meets expectations.

Where Admission Actually Runs and Where Bypasses Remain

Admission evaluates API requests that match its configuration. It is not a continuous reevaluation of everything running. For a webhook, the scope must enumerate resources and operations, API versions, matchPolicy, namespaceSelector, objectSelector, and matchConditions. A request excluded by a selector is never evaluated, irrespective of failurePolicy.

failurePolicy has a narrower meaning than many designs assume. Fail rejects a matching request when the webhook call errors or times out; Ignore permits the API server to continue past that call failure. If the webhook successfully receives the request and returns allowed: false, that is an explicit rejection and Ignore does not turn it into an approval. Conversely, Fail cannot protect a request that never matched the configured rules, API version, or selectors.

The test set cannot stop at direct Pod creation. Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs hold Pod templates, and controllers create Pods later. A particular policy engine may check the high-level resource, the resulting Pod, or both. Acceptance must show the expected behavior for every permitted create and update path, including init containers, long-running sidecars, debugging ephemeral containers, and supported image volumes. If the engine does not inspect one of them, that is a documented boundary.

The “no policy matched” behavior also requires a negative test. Allow, warn, and deny provide different guarantees. Warning is useful during rollout but does not enforce a release decision. Opt-in namespaces need their own test: a newly created namespace without the label must not silently become a production bypass.

An Admitted Request Does Not Prove Current Runtime State

Admission answers a question about one operation at one time. It does not automatically reevaluate a Pod created yesterday when a digest is denied today or a signing identity loses trust. Runtime assurance therefore requires a separate reconciliation process.

A minimum inventory row contains cluster, namespace, workload and owner, Pod name, container and its role, reference from the specification, admitted digest, runtime-reported image identity, node platform, start time, and the policy version in force at admission. It includes application, init, and persistent sidecar containers; debugging containers are recorded separately. The snapshot should connect to the release record, and every discrepancy should have an owner and deadline. Check freshness separately: Pod status can lag, particularly when a node loses contact with the control plane. Acceptance should require confirmation of that node’s state and mapping its image identity to the digest chain, rather than treating a stale status as verified runtime evidence.

Even perfect digest agreement does not establish that every byte used by the process remains immutable after start. Image layers are read-only, but the container can have a writable layer and the Pod can mount emptyDir, persistent volumes, ConfigMaps, Secrets, and other data. The application can also download and execute content after startup. Those bytes do not become part of the main image digest. Where runtime retrieval is permitted, policy must govern source, integrity, and execution separately. Where it is prohibited, test the application behavior and egress enforcement. The bounded claim matters: image signing verifies the image object, not every later state of the process.

Withdrawing an Image Without Unapproved Downtime

Adding a digest to a deny list, removing trust in a signer, or tightening provenance expectations should block future matching admission operations. Those changes do not terminate containers already running. When the kubelet restarts a container under restartPolicy inside the same Pod on the same node, it does not create a new Pod API object and there is no new Pod admission. A deny list alone therefore cannot guarantee that withdrawn bytes will never start again. Deleting all affected Pods automatically is also not a safe universal answer: quorum services, singleton components, and StatefulSets can turn a security response into an availability incident.

A withdrawal plan starts with two inventories: where the digest actually runs and which approved digest can replace it. The risk owner then selects a contextual action: immediate isolation for active compromise, a controlled rollout for a vulnerable image without evidence of exploitation, or a short exception with compensating controls for a component that cannot yet be replaced. The plan separately covers a container restart inside an existing Pod, a replacement Pod created by a controller, and excluded or static paths. Lack of a replacement should surface as a risk decision, rather than silently becoming the decision.

For a Deployment, update availability depends on the rollout strategy, maxUnavailable, maxSurge, readiness checks, and the new version’s ability to serve. A PodDisruptionBudget constrains voluntary evictions, but it does not by itself govern a workload controller’s rolling update. When responders withdraw instances through the Eviction API, the PDB and grace period participate; direct mass deletion skips useful coordination. A StatefulSet additionally needs an order, data-safety constraints, and quorum rules. A DaemonSet needs a minimum safe node set. A Job needs retry and idempotency decisions.

Rollback may target only a known digest that still satisfies the current policy. Rolling back to a previous tag can select a different object if the tag moved. Verify the fallback digest through the same admission path first, start a canary or a small wave, evaluate health and business signals, and only then widen the rollout. Deleting a withdrawn object from the registry before replacement completes can prevent new Pods from starting while old ones continue to run, creating a false impression that withdrawal succeeded.

An Exception Is a Bounded Decision, Not a Permanent Tunnel

System add-ons, emergency releases, and vendor images sometimes cannot satisfy the common policy immediately. An exception is safer than disabling enforcement broadly only when it is narrow and machine-readable. Its record needs an exact digest, cluster and namespace, resource kinds, reason, owner, risk approver, start and expiry, compensating controls, and a tested exit path.

An exception for a tag or an entire registry is too broad because its future meaning can change without an edit to the exception. Signer identity regular expressions need negative test cases. A break-glass path should leave two separate records: who authorized the bypass and who applied it, followed by reconciliation of every Pod created during the window. Expiry should close the exception automatically or produce a blocking signal to the owner; an unattended calendar reminder is weak enforcement.

Exceptions belong in withdrawal exercises too. The team must know whether a deny list overrides the exception, who can extend it, and whether the workload has a safe replacement path. Otherwise the most critical component can sit outside the controlled process precisely when an incident occurs.

What to Procure and What an Authorized Pentest May Do

A buyer needs a reproducible evidence pack, rather than a screenshot of a successful verification. The statement of work should request a registry and cluster map, export of versioned policies, rule-matching matrix, test digests and platform mapping, positive and negative test results, desired-versus-running image inventory, exception register, withdrawal runbook, and a recorded rollback exercise. Each report assertion should identify whether it was confirmed by a test, inferred from configuration, or supplied by an owner.

Authorized testing can attempt bypasses through another workload kind or API version, an unlabeled namespace, a nonmatching object selector, an image with no matching policy, webhook timeout, wrong signer subject or issuer, a signature by the right identity over the wrong digest, an attestation with an unexpected buildType, an alternate platform manifest, and a discrepancy between the PodSpec and runtime inventory. A separate lifecycle test terminates a container and observes whether the kubelet can restart the same image inside the existing Pod without a new admission. Outage cases require an agreed test environment or window because fail-closed can halt legitimate deployment while fail-open can admit an unverified request.

The authorization must say whether the tester may create workloads, degrade or disable a webhook, submit malformed attestations, read registry metadata, terminate a container to test restart behavior, or trigger a controlled rollout. Deleting production images, stopping Pods in bulk, accessing private signing keys, and loading the API server do not follow automatically from the word “pentest.” A managed cluster or third-party registry may also require its owner’s separate permission.

Pentest.red can help define this technical boundary through the pentest scoping form: clusters, registries, permitted bypass cases, test digests, maintenance windows, and stop conditions. The penetration testing service page explains the engagement format, while the sample pentest report helps agree on evidence before work begins. The result is technical findings within an authorized sample and environment. It is not a certification or proof that every untested image is secure.

Acceptance Matrix: Assertions That Need Evidence

Complete this matrix for each production cluster and each release path. “Pass” means a reproducible artifact was retained, rather than a team member saying the control worked.

Findings should become testable engineering work with an owner, exact object, completion condition, and retest. The penetration test remediation validation guide develops that acceptance discipline further.

The Readiness Decision: Four Questions for the Release Owner

Before release, the owner should be able to name the exact index and platform-manifest digests; show why the signer, issuer, builder, and parameters are permitted; reproduce a negative test for every workload creation path; and produce a dated inventory of what actually runs. Before withdrawal, add an approved replacement, availability-impact assessment, a same-Pod restart test, rollout waves, and a stop criterion.

A missing answer does not always require an immediate business shutdown. It means the decision lacks evidence and must be treated as a bounded exception with an owner and expiry. That is how container infrastructure security becomes an operating decision system: the organization can prove what it admitted, find where it runs, and replace a withdrawn release without improvising during an incident.