Home/For responders/Checkpoints and node memory
Checkpoints and node memory
There are two ways to keep what a container held in memory. A checkpoint copies one container's memory and files while it keeps running. A node memory image copies everything the node's kernel held, every container on it included. Which one works depends on the cloud, the node image, and the container runtime. Node memory works on more of them.
Which one you can do
| Nodes | Checkpoint one container | Capture node memory | Symbols for Volatility |
|---|---|---|---|
| EKS, Amazon Linux 2023 | Yes, once CRIU is in the node image. AMIs for Kubernetes 1.30 and later moved to containerd 2 in late 2025 | Yes, with AVML through SSM Run Command | The amazonlinux-debuginfo repository, off by default |
| EKS, Bottlerocket | No. There is no CRIU package | Probably, with AVML from the admin container. Not tested by AWS | None published. The kernel's BTF data is a partial route |
| EKS Auto Mode, Fargate | No | No. Auto Mode has no SSM and no host containers; Fargate has no node to reach | |
| GKE Standard, Container-Optimized OS | Pod snapshots, for pods already running in GKE Sandbox. The kubelet route fails: CRIU is absent and the root filesystem is read-only | Yes, with AVML from a privileged pod on the node | gs://cos-tools/<BUILD_ID>/vmlinux |
| GKE Standard, Ubuntu | Not tested. GKE 1.33 and later ship containerd 2; CRIU has to be added to the host | Yes, the same way as on Container-Optimized OS | Ubuntu's debug symbol archive, which does not always carry the -gke kernels |
| GKE Autopilot | Only Pod snapshots, for pods already running in GKE Sandbox on GKE 1.35.3 or later | No. Privileged pods are refused |
A checkpoint through the kubelet needs four things on the node. The first is the kubelet's checkpoint API, beta and on by default since Kubernetes 1.30. The second is a runtime that implements it: CRI-O, or containerd 2.0 and later. The third is the CRIU tool, and the fourth a credential the kubelet accepts. containerd 1.7 answers that the method is not implemented.
Before an incident
- Find out which row each node pool is in, and run both procedures below on a scratch node. A runbook that has never been run fails at step one.
- On EKS with Amazon Linux 2023, install CRIU in the node
image or launch template. Installing it during an incident writes to the disk
you are about to image. Add
AmazonSSMManagedInstanceCoreto the node role. - Give the node, or a responder credential,
s3:PutObjecton one prefix of one evidence bucket and nothing else. Turn on Object Lock, so an attacker on the node cannot overwrite or delete what was written. - Grant a break-glass identity
createonnodes/checkpoint, and nothing wider. The common shortcut,nodes/proxy, reaches every kubelet API, exec in any container included. GKE Autopilot refuses to grant it at all. - Stage the tools. Keep a copy of AVML in your own bucket or
registry, pinned by SHA-256, and an analysis machine with Volatility 3 and
dwarf2json. Build symbols for every kernel you run before you need them: Volatility matches a symbol table to the image's kernel banner exactly, build date included. - On GKE, a workload you will want to snapshot has to run in
GKE Sandbox before the incident. It cannot be moved there afterwards without
restarting it. The cluster needs Pod snapshots switched on with
--enable-pod-snapshots, and aPodSnapshotStorageConfignaming a bucket in the cluster's location. The bucket needs hierarchical namespace and uniform bucket-level access, with soft delete off.
Runbook: checkpoint one container on EKS
- Keep the node as container forensics describes: cordon it, and set termination and scale-in protection.
- Check CRIU on the node, through SSM:
sudo criu check
- Get a short token for the break-glass service account. Ten minutes is the shortest allowed, and only deleting the service account revokes it:
kubectl create token checkpoint-responder -n kube-system --duration=10m
- Ask the kubelet, from the node, for the container's checkpoint. The container keeps running:
curl -sk -X POST -H "Authorization: Bearer $TOKEN" \
"https://localhost:10250/checkpoint/<namespace>/<pod>/<container>?timeout=120"
A success returns the archive's path, under
/var/lib/kubelet/checkpoints/, named for the pod, namespace,
container, and time. A 404 means the feature is off or a name is wrong. A 500
means the runtime cannot do it: containerd 1.7, or no CRIU.
- Hash it and move it. The archive is written to the node's disk, readable by root, and as large as the container's memory:
sha256sum /var/lib/kubelet/checkpoints/checkpoint-*.tar | tee checkpoint.sha256
aws s3 cp /var/lib/kubelet/checkpoints/<archive>.tar s3://<evidence-bucket>/<case>/
aws s3 cp checkpoint.sha256 s3://<evidence-bucket>/<case>/
The archive contains every memory page of every process in the container, keys and passwords included. Store it like the most sensitive evidence you have. Writing it may overwrite deleted files the attacker left, and the disk snapshot you take next will contain the archive, secrets and all, even after it is deleted. Take the snapshot, then delete the archive.
Reading a checkpoint
| Question | Command |
|---|---|
| What is in the archive | checkpointctl show <archive>.tar |
| Processes, with their commands and environment | checkpointctl inspect --ps-tree --ps-tree-cmd --ps-tree-env <archive>.tar |
| Open files and sockets | checkpointctl inspect --files --sockets <archive>.tar |
| Memory, per process | checkpointctl memparse --pid <pid> --search-regex '<pattern>' --context 64 <archive>.tar |
| Files the container wrote or changed | rootfs-diff.tar inside the archive, and deleted.files on CRI-O |
To watch it run again, checkpointctl build turns the archive into
an image that restores the container. Restore it only in an isolated cluster
with no route out, on CRI-O or containerd 2.1 and later. It resumes with the
attacker's code and connections as they were.
On GKE: Pod snapshots
GKE's documented way to save a pod's memory is a Pod snapshot. It saves
process memory, threads, and filesystem changes to Cloud Storage. It applies only
to pods in GKE Sandbox, on GKE 1.35.3 or later, and it is started by creating a
PodSnapshotManualTrigger naming the pod. Google's documentation
says GKE pauses the pod while it takes the snapshot. Lock the bucket down: every
snapshot contains the pod's secrets.
Runbook: node memory on EKS with Amazon Linux 2023
Stream the image straight to S3, so nothing is written to the node's disk, then snapshot the disks. AWS's EKS best practices guide puts memory before the disk snapshot too.
- Keep the node: cordon, termination protection, scale-in protection. Spot instances cannot take termination protection, so work quickly.
- Run AVML through SSM with the
AWS-RunShellScriptdocument and a timeout of an hour or more. AVML refuses to write to/dev/stdout, so it writes into a named pipe that the upload reads from:
set -euo pipefail
cd /run
aws s3 cp s3://<tools-bucket>/avml-0.20.0 ./avml
echo "a67010b08af2edf5a04d2f051a133e0d8e30853c308f055b8335e759609d6dc9 avml" | sha256sum -c
chmod +x avml
uname -r | aws s3 cp - s3://<evidence-bucket>/<case>/<node>.kernel.txt
test -s /proc/kcore || { echo "/proc/kcore is not readable"; exit 1; }
mkfifo /run/avml.pipe
tee >(sha256sum > /run/<node>.lime.sha256) < /run/avml.pipe \
| aws s3 cp - s3://<evidence-bucket>/<case>/<node>.lime \
--expected-size "$(awk '/MemTotal/ {print int($2 * 1024 * 1.1)}' /proc/meminfo)" &
timeout 3h ./avml acquire /run/avml.pipe
wait
sleep 2
aws s3 cp /run/<node>.lime.sha256 s3://<evidence-bucket>/<case>/
The hash above is for AVML 0.20.0 on x86_64. The image is about as large as
the node's memory and a little larger, so --expected-size adds a
tenth: it lets the upload pick a part size that fits S3's limit. If
/run is mounted noexec, use /root and note
the small write. If AVML fails before it opens the pipe, the upload waits on it,
and the timeout ends both.
- Check the image on the analysis machine. Its hash must match, and its kernel banner must match the kernel you recorded:
sha256sum <node>.lime # compare with <node>.lime.sha256
vol -f <node>.lime banners.Banners
- Snapshot the node's EBS volumes, then quarantine.
- Build symbols on an Amazon Linux 2023 analysis machine on
the same release as the node. Install the debuginfo package for that exact
kernel from the debuginfo repository, which is off by default. Then run
dwarf2json linux --elf <vmlinux> > <kernel>.json, which needs about 8 GB of memory.
Node memory on Bottlerocket
Bottlerocket runs SSM in its control container, which is on by default. The
admin container is off until you enable it with enable-admin-container,
and sheltie inside it gives a root shell on the host. Bottlerocket
locks the kernel down in integrity mode, which still leaves
/proc/kcore readable, so AVML should work from there. Nobody has
published a tested run, so test it on a scratch node first.
Bottlerocket publishes no debug symbols. The kernel carries BTF type data at
/sys/kernel/btf/vmlinux, and the community tool
btf2json builds a partial symbol table from it and a symbol map.
Copy /sys/kernel/btf/vmlinux and /proc/kallsyms off
the node at capture time, since nothing publishes them later. Volatility accepts
the result only with the patch btf2json documents, and some plugins
will still fail.
Never lower the lockdown setting to capture memory. It takes a reboot, and the reboot destroys the memory.
Runbook: node memory on GKE with Container-Optimized OS
- Keep the node: annotate it so the autoscaler leaves it, and cordon it. Auto-repair replaces a node that stays NotReady for about 10 minutes, so move straight on.
- Start a privileged pod on the node from your own image with
AVML and
timeoutin it, pinned by digest. Thesysadminprofile makes it privileged, which unmasks/proc/kcore:
kubectl debug node/<node> -it --profile=sysadmin \
--image=<registry>/ir-tools@sha256:<digest> -- sh
grep BUILD_ID /host/etc/os-release # the Container-Optimized OS build
uname -r
- Stream the image to your machine through the API server, so nothing lands on the node's disk. In a second terminal:
kubectl exec -i <debug-pod> -- sh -c \
'mkfifo /dev/shm/m && { timeout 3h /avml acquire /dev/shm/m >&2 & } && cat /dev/shm/m' \
| tee >(sha256sum > <node>.lime.sha256) > <node>.lime
AVML refuses to write to /dev/stdout, so it writes into a named
pipe in memory, and cat sends the pipe out. The debug pod is named
node-debugger-<node>- and five characters. This moves the
whole of the node's memory through the API server, which is slow for a large
node. Test the speed before an incident. The alternative is to
snapshot the node's disks first, then write the image to
/host/var and upload it with a signed URL. GKE nodes cannot write
to Cloud Storage by default.
- Snapshot the node's disks, delete the debug pod, then quarantine.
- Fetch the kernel for that build and build symbols:
gcloud storage cp gs://cos-tools/<BUILD_ID>/vmlinux . # Arm: <BUILD_ID>/lakitu-arm64/vmlinux
dwarf2json linux --elf vmlinux > cos-<BUILD_ID>.json
If AVML reports that /proc/kcore is locked down, the node runs
in confidentiality mode and its memory cannot be read without a reboot. Rely on
the disk and the logs.
Reading node memory
- Run
vol -f <image> banners.Bannersand confirm the kernel banner matches the one you recorded. - Copy the symbol table into
volatility3/symbols/linux/, or pass its folder with-s.vol isfinfolists the banners Volatility can match. - Find the container's processes. Each pod runs under one
containerd-shim-runc-v2process, whose children are the pod's pause process and the first process of each container.
| Question | Plugin |
|---|---|
| Which processes belong to which pod | linux.pstree: children of each containerd-shim-runc-v2 |
| Which pod a process is in | linux.envars --pid <pid>: HOSTNAME is the pod's name |
| Its command line | linux.psaux |
| Its filesystem view | linux.mountinfo --pids <pid>, which shows the mount namespace |
| Its connections | linux.sockstat --pids <pid>, with the network namespace |
| Code injected into memory | linux.malware.malfind |
| Files it held open | linux.lsof --pid <pid> |
Volatility has no plugin for cgroups, so the pod name in the environment and the mount namespace are the ties from a process to a container.
Cautions
- A node image is as large as the node's memory, and taking it takes minutes to tens of minutes. Every container on the node is in it.
- Hash while you stream, and record the hash of the AVML binary you ran.
- Writing a memory image or a checkpoint to the node's own disk changes the disk evidence. Stream it off, or snapshot the disk first.
- Both a checkpoint and a node image contain secrets in clear: tokens, keys, passwords. Encrypt them and limit who can read them.
The container evidence template keeps the node and records the pod; these runbooks add memory to it.