Home/For responders/Container forensics
Container forensics
A container is a process on a node with its own view of the filesystem. Its evidence sits in a few places, and each place disappears at a different moment.
Where a container keeps its evidence
| Evidence | Where it lives | Gone when |
|---|---|---|
| The pod's definition | The cluster API: image, service account, volumes, security settings | The pod is deleted |
| The image | Read-only layers on the node and in the registry, named by digest | Rarely. A tag can be moved to other code, a digest cannot |
| The writable layer | A directory on the node containing every file the container wrote or changed | The container restarts |
| Memory | The node's memory, since a container's processes run on the node's kernel | The container stops |
| Logs | Files on the node, rotated at a size limit, and wherever you ship them | Rotation, or the pod being removed from the node |
emptyDir volumes | The node's disk | The pod is removed from the node |
| Persistent volumes | A cloud disk attached to the node | Only when the volume is deleted |
| What was done to it | The cluster audit log and the cloud's audit log | Their retention runs out |
Kubernetes restarts containers on its own, so the writable layer and memory
can vanish without anyone deciding anything. Logs rotate by size, by default at
10 MiB with five files kept on GKE. kubectl logs returns only the
current file, and --previous the one from before the last restart.
The order to capture it in
- Record the pod. Its full definition, and each image by the
digest in
imageID. This reads the API and changes nothing. - Save the logs, current and previous, before they rotate.
- Keep the node. Stop the autoscaler removing it, cordon it, and on AWS protect the instance from termination and scale-in.
- Checkpoint the container, where your cluster supports it. That captures memory and the writable layer together.
- Snapshot the node's disks. The writable layer, the logs, and
any
emptyDirare on them. - Cut the pod off with a quarantine label and a deny-all network policy.
- Only then look inside, knowing that looking changes things.
The container evidence template does steps 1, 2, 3, 5, and 6 on GKE or EKS, and hashes every file it collects.
Checkpointing a container
The kubelet's checkpoint API, beta and on by default since Kubernetes 1.30,
copies a running container's memory and filesystem changes into an archive on the
node without stopping it. Send POST /checkpoint/<namespace>/<pod>/<container>
to the kubelet, and the archive appears under /var/lib/kubelet/checkpoints.
- The container runtime must support it, and the CRIU tool must be on the node. CRI-O does. containerd 1.7 does not, and containerd 2 does.
- The archive contains every memory page, keys and passwords included. Store it like the most sensitive evidence you have.
- Reaching the kubelet directly needs rights close to control of the node. Keep them for the responders' emergency path, and test it before an incident.
Checkpoints and node memory has the step-by-step runbooks for EKS and GKE, and which nodes support which.
Looking inside without changing much
| Tool | What it gives you | What it changes |
|---|---|---|
kubectl debug -it <pod> --image=<tools> --target=<container> | A shell in an ephemeral container that sees the target's processes | Adds a container to the pod for the rest of its life. It cannot be removed |
kubectl debug node/<node> -it --image=<tools> | A pod in the node's process, network, and IPC namespaces, with the node's filesystem at /host | Schedules a new pod on the node. Delete it afterwards. It needs --profile=sysadmin to act as root on the host |
crictl ps, crictl inspect, crictl logs | The runtime's view: containers, their mounts and process IDs, their logs | Nothing, once you are on the node |
/proc/<pid>/root on the node | The container's filesystem, as its process sees it | Nothing, if you only read |
With containerd's default overlay filesystem, the writable layer is a directory
under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/.
Compare it with the image: anything added under /tmp, /dev/shm,
or a binary directory is where most intrusions show.
Moves that destroy container evidence
kubectl delete pod, or a rollout restart of its deployment.- Scaling the deployment to zero to "stop the bleeding".
- Copying tools into the container with
kubectl cporexec: they land in the writable layer you are trying to preserve, and whoever is inside may be watching. - Letting the autoscaler or node repair replace the node.
- Rebuilding from the same tag, which may pull different code than what was running. Record the digest first.
Reading what you captured
- The definition: privileged mode, host paths, host namespaces, and the service account. Each one is a path out of the container, listed in Kubernetes for defenders.
- The writable layer: new files, and changed files that should never change at runtime.
- The audit log: who created the pod, who ran exec in it, and what its service account called.
- Runtime alerts: Falco or your cloud's container threat detection, for what ran inside.