iac.htora.dev · security templates

Home/For responders/Container forensics

Container forensics

A container is a process on a node with its own view of the filesystem. Its evidence sits in a few places, and each place disappears at a different moment.

Where a container keeps its evidence

EvidenceWhere it livesGone when
The pod's definitionThe cluster API: image, service account, volumes, security settingsThe pod is deleted
The imageRead-only layers on the node and in the registry, named by digestRarely. A tag can be moved to other code, a digest cannot
The writable layerA directory on the node containing every file the container wrote or changedThe container restarts
MemoryThe node's memory, since a container's processes run on the node's kernelThe container stops
LogsFiles on the node, rotated at a size limit, and wherever you ship themRotation, or the pod being removed from the node
emptyDir volumesThe node's diskThe pod is removed from the node
Persistent volumesA cloud disk attached to the nodeOnly when the volume is deleted
What was done to itThe cluster audit log and the cloud's audit logTheir retention runs out

Kubernetes restarts containers on its own, so the writable layer and memory can vanish without anyone deciding anything. Logs rotate by size, by default at 10 MiB with five files kept on GKE. kubectl logs returns only the current file, and --previous the one from before the last restart.

The order to capture it in

  1. Record the pod. Its full definition, and each image by the digest in imageID. This reads the API and changes nothing.
  2. Save the logs, current and previous, before they rotate.
  3. Keep the node. Stop the autoscaler removing it, cordon it, and on AWS protect the instance from termination and scale-in.
  4. Checkpoint the container, where your cluster supports it. That captures memory and the writable layer together.
  5. Snapshot the node's disks. The writable layer, the logs, and any emptyDir are on them.
  6. Cut the pod off with a quarantine label and a deny-all network policy.
  7. Only then look inside, knowing that looking changes things.

The container evidence template does steps 1, 2, 3, 5, and 6 on GKE or EKS, and hashes every file it collects.

Checkpointing a container

The kubelet's checkpoint API, beta and on by default since Kubernetes 1.30, copies a running container's memory and filesystem changes into an archive on the node without stopping it. Send POST /checkpoint/<namespace>/<pod>/<container> to the kubelet, and the archive appears under /var/lib/kubelet/checkpoints.

  • The container runtime must support it, and the CRIU tool must be on the node. CRI-O does. containerd 1.7 does not, and containerd 2 does.
  • The archive contains every memory page, keys and passwords included. Store it like the most sensitive evidence you have.
  • Reaching the kubelet directly needs rights close to control of the node. Keep them for the responders' emergency path, and test it before an incident.

Checkpoints and node memory has the step-by-step runbooks for EKS and GKE, and which nodes support which.

Looking inside without changing much

ToolWhat it gives youWhat it changes
kubectl debug -it <pod> --image=<tools> --target=<container>A shell in an ephemeral container that sees the target's processesAdds a container to the pod for the rest of its life. It cannot be removed
kubectl debug node/<node> -it --image=<tools>A pod in the node's process, network, and IPC namespaces, with the node's filesystem at /hostSchedules a new pod on the node. Delete it afterwards. It needs --profile=sysadmin to act as root on the host
crictl ps, crictl inspect, crictl logsThe runtime's view: containers, their mounts and process IDs, their logsNothing, once you are on the node
/proc/<pid>/root on the nodeThe container's filesystem, as its process sees itNothing, if you only read

With containerd's default overlay filesystem, the writable layer is a directory under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/. Compare it with the image: anything added under /tmp, /dev/shm, or a binary directory is where most intrusions show.

Moves that destroy container evidence

  • kubectl delete pod, or a rollout restart of its deployment.
  • Scaling the deployment to zero to "stop the bleeding".
  • Copying tools into the container with kubectl cp or exec: they land in the writable layer you are trying to preserve, and whoever is inside may be watching.
  • Letting the autoscaler or node repair replace the node.
  • Rebuilding from the same tag, which may pull different code than what was running. Record the digest first.

Reading what you captured

  • The definition: privileged mode, host paths, host namespaces, and the service account. Each one is a path out of the container, listed in Kubernetes for defenders.
  • The writable layer: new files, and changed files that should never change at runtime.
  • The audit log: who created the pod, who ran exec in it, and what its service account called.
  • Runtime alerts: Falco or your cloud's container threat detection, for what ran inside.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.