iac.htora.dev · security templates

Home/For responders/Checkpoints and node memory

Checkpoints and node memory

There are two ways to keep what a container held in memory. A checkpoint copies one container's memory and files while it keeps running. A node memory image copies everything the node's kernel held, every container on it included. Which one works depends on the cloud, the node image, and the container runtime. Node memory works on more of them.

Which one you can do

NodesCheckpoint one containerCapture node memorySymbols for Volatility
EKS, Amazon Linux 2023Yes, once CRIU is in the node image. AMIs for Kubernetes 1.30 and later moved to containerd 2 in late 2025Yes, with AVML through SSM Run CommandThe amazonlinux-debuginfo repository, off by default
EKS, BottlerocketNo. There is no CRIU packageProbably, with AVML from the admin container. Not tested by AWSNone published. The kernel's BTF data is a partial route
EKS Auto Mode, FargateNoNo. Auto Mode has no SSM and no host containers; Fargate has no node to reach
GKE Standard, Container-Optimized OSPod snapshots, for pods already running in GKE Sandbox. The kubelet route fails: CRIU is absent and the root filesystem is read-onlyYes, with AVML from a privileged pod on the nodegs://cos-tools/<BUILD_ID>/vmlinux
GKE Standard, UbuntuNot tested. GKE 1.33 and later ship containerd 2; CRIU has to be added to the hostYes, the same way as on Container-Optimized OSUbuntu's debug symbol archive, which does not always carry the -gke kernels
GKE AutopilotOnly Pod snapshots, for pods already running in GKE Sandbox on GKE 1.35.3 or laterNo. Privileged pods are refused

A checkpoint through the kubelet needs four things on the node. The first is the kubelet's checkpoint API, beta and on by default since Kubernetes 1.30. The second is a runtime that implements it: CRI-O, or containerd 2.0 and later. The third is the CRIU tool, and the fourth a credential the kubelet accepts. containerd 1.7 answers that the method is not implemented.

Before an incident

  • Find out which row each node pool is in, and run both procedures below on a scratch node. A runbook that has never been run fails at step one.
  • On EKS with Amazon Linux 2023, install CRIU in the node image or launch template. Installing it during an incident writes to the disk you are about to image. Add AmazonSSMManagedInstanceCore to the node role.
  • Give the node, or a responder credential, s3:PutObject on one prefix of one evidence bucket and nothing else. Turn on Object Lock, so an attacker on the node cannot overwrite or delete what was written.
  • Grant a break-glass identity create on nodes/checkpoint, and nothing wider. The common shortcut, nodes/proxy, reaches every kubelet API, exec in any container included. GKE Autopilot refuses to grant it at all.
  • Stage the tools. Keep a copy of AVML in your own bucket or registry, pinned by SHA-256, and an analysis machine with Volatility 3 and dwarf2json. Build symbols for every kernel you run before you need them: Volatility matches a symbol table to the image's kernel banner exactly, build date included.
  • On GKE, a workload you will want to snapshot has to run in GKE Sandbox before the incident. It cannot be moved there afterwards without restarting it. The cluster needs Pod snapshots switched on with --enable-pod-snapshots, and a PodSnapshotStorageConfig naming a bucket in the cluster's location. The bucket needs hierarchical namespace and uniform bucket-level access, with soft delete off.

Runbook: checkpoint one container on EKS

  1. Keep the node as container forensics describes: cordon it, and set termination and scale-in protection.
  2. Check CRIU on the node, through SSM:
sudo criu check
  1. Get a short token for the break-glass service account. Ten minutes is the shortest allowed, and only deleting the service account revokes it:
kubectl create token checkpoint-responder -n kube-system --duration=10m
  1. Ask the kubelet, from the node, for the container's checkpoint. The container keeps running:
curl -sk -X POST -H "Authorization: Bearer $TOKEN" \
  "https://localhost:10250/checkpoint/<namespace>/<pod>/<container>?timeout=120"

A success returns the archive's path, under /var/lib/kubelet/checkpoints/, named for the pod, namespace, container, and time. A 404 means the feature is off or a name is wrong. A 500 means the runtime cannot do it: containerd 1.7, or no CRIU.

  1. Hash it and move it. The archive is written to the node's disk, readable by root, and as large as the container's memory:
sha256sum /var/lib/kubelet/checkpoints/checkpoint-*.tar | tee checkpoint.sha256
aws s3 cp /var/lib/kubelet/checkpoints/<archive>.tar s3://<evidence-bucket>/<case>/
aws s3 cp checkpoint.sha256 s3://<evidence-bucket>/<case>/

The archive contains every memory page of every process in the container, keys and passwords included. Store it like the most sensitive evidence you have. Writing it may overwrite deleted files the attacker left, and the disk snapshot you take next will contain the archive, secrets and all, even after it is deleted. Take the snapshot, then delete the archive.

Reading a checkpoint

QuestionCommand
What is in the archivecheckpointctl show <archive>.tar
Processes, with their commands and environmentcheckpointctl inspect --ps-tree --ps-tree-cmd --ps-tree-env <archive>.tar
Open files and socketscheckpointctl inspect --files --sockets <archive>.tar
Memory, per processcheckpointctl memparse --pid <pid> --search-regex '<pattern>' --context 64 <archive>.tar
Files the container wrote or changedrootfs-diff.tar inside the archive, and deleted.files on CRI-O

To watch it run again, checkpointctl build turns the archive into an image that restores the container. Restore it only in an isolated cluster with no route out, on CRI-O or containerd 2.1 and later. It resumes with the attacker's code and connections as they were.

On GKE: Pod snapshots

GKE's documented way to save a pod's memory is a Pod snapshot. It saves process memory, threads, and filesystem changes to Cloud Storage. It applies only to pods in GKE Sandbox, on GKE 1.35.3 or later, and it is started by creating a PodSnapshotManualTrigger naming the pod. Google's documentation says GKE pauses the pod while it takes the snapshot. Lock the bucket down: every snapshot contains the pod's secrets.

Runbook: node memory on EKS with Amazon Linux 2023

Stream the image straight to S3, so nothing is written to the node's disk, then snapshot the disks. AWS's EKS best practices guide puts memory before the disk snapshot too.

  1. Keep the node: cordon, termination protection, scale-in protection. Spot instances cannot take termination protection, so work quickly.
  2. Run AVML through SSM with the AWS-RunShellScript document and a timeout of an hour or more. AVML refuses to write to /dev/stdout, so it writes into a named pipe that the upload reads from:
set -euo pipefail
cd /run
aws s3 cp s3://<tools-bucket>/avml-0.20.0 ./avml
echo "a67010b08af2edf5a04d2f051a133e0d8e30853c308f055b8335e759609d6dc9  avml" | sha256sum -c
chmod +x avml
uname -r | aws s3 cp - s3://<evidence-bucket>/<case>/<node>.kernel.txt
test -s /proc/kcore || { echo "/proc/kcore is not readable"; exit 1; }
mkfifo /run/avml.pipe
tee >(sha256sum > /run/<node>.lime.sha256) < /run/avml.pipe \
  | aws s3 cp - s3://<evidence-bucket>/<case>/<node>.lime \
      --expected-size "$(awk '/MemTotal/ {print int($2 * 1024 * 1.1)}' /proc/meminfo)" &
timeout 3h ./avml acquire /run/avml.pipe
wait
sleep 2
aws s3 cp /run/<node>.lime.sha256 s3://<evidence-bucket>/<case>/

The hash above is for AVML 0.20.0 on x86_64. The image is about as large as the node's memory and a little larger, so --expected-size adds a tenth: it lets the upload pick a part size that fits S3's limit. If /run is mounted noexec, use /root and note the small write. If AVML fails before it opens the pipe, the upload waits on it, and the timeout ends both.

  1. Check the image on the analysis machine. Its hash must match, and its kernel banner must match the kernel you recorded:
sha256sum <node>.lime        # compare with <node>.lime.sha256
vol -f <node>.lime banners.Banners
  1. Snapshot the node's EBS volumes, then quarantine.
  2. Build symbols on an Amazon Linux 2023 analysis machine on the same release as the node. Install the debuginfo package for that exact kernel from the debuginfo repository, which is off by default. Then run dwarf2json linux --elf <vmlinux> > <kernel>.json, which needs about 8 GB of memory.

Node memory on Bottlerocket

Bottlerocket runs SSM in its control container, which is on by default. The admin container is off until you enable it with enable-admin-container, and sheltie inside it gives a root shell on the host. Bottlerocket locks the kernel down in integrity mode, which still leaves /proc/kcore readable, so AVML should work from there. Nobody has published a tested run, so test it on a scratch node first.

Bottlerocket publishes no debug symbols. The kernel carries BTF type data at /sys/kernel/btf/vmlinux, and the community tool btf2json builds a partial symbol table from it and a symbol map. Copy /sys/kernel/btf/vmlinux and /proc/kallsyms off the node at capture time, since nothing publishes them later. Volatility accepts the result only with the patch btf2json documents, and some plugins will still fail.

Never lower the lockdown setting to capture memory. It takes a reboot, and the reboot destroys the memory.

Runbook: node memory on GKE with Container-Optimized OS

  1. Keep the node: annotate it so the autoscaler leaves it, and cordon it. Auto-repair replaces a node that stays NotReady for about 10 minutes, so move straight on.
  2. Start a privileged pod on the node from your own image with AVML and timeout in it, pinned by digest. The sysadmin profile makes it privileged, which unmasks /proc/kcore:
kubectl debug node/<node> -it --profile=sysadmin \
  --image=<registry>/ir-tools@sha256:<digest> -- sh
grep BUILD_ID /host/etc/os-release      # the Container-Optimized OS build
uname -r
  1. Stream the image to your machine through the API server, so nothing lands on the node's disk. In a second terminal:
kubectl exec -i <debug-pod> -- sh -c \
  'mkfifo /dev/shm/m && { timeout 3h /avml acquire /dev/shm/m >&2 & } && cat /dev/shm/m' \
  | tee >(sha256sum > <node>.lime.sha256) > <node>.lime

AVML refuses to write to /dev/stdout, so it writes into a named pipe in memory, and cat sends the pipe out. The debug pod is named node-debugger-<node>- and five characters. This moves the whole of the node's memory through the API server, which is slow for a large node. Test the speed before an incident. The alternative is to snapshot the node's disks first, then write the image to /host/var and upload it with a signed URL. GKE nodes cannot write to Cloud Storage by default.

  1. Snapshot the node's disks, delete the debug pod, then quarantine.
  2. Fetch the kernel for that build and build symbols:
gcloud storage cp gs://cos-tools/<BUILD_ID>/vmlinux .   # Arm: <BUILD_ID>/lakitu-arm64/vmlinux
dwarf2json linux --elf vmlinux > cos-<BUILD_ID>.json

If AVML reports that /proc/kcore is locked down, the node runs in confidentiality mode and its memory cannot be read without a reboot. Rely on the disk and the logs.

Reading node memory

  1. Run vol -f <image> banners.Banners and confirm the kernel banner matches the one you recorded.
  2. Copy the symbol table into volatility3/symbols/linux/, or pass its folder with -s. vol isfinfo lists the banners Volatility can match.
  3. Find the container's processes. Each pod runs under one containerd-shim-runc-v2 process, whose children are the pod's pause process and the first process of each container.
QuestionPlugin
Which processes belong to which podlinux.pstree: children of each containerd-shim-runc-v2
Which pod a process is inlinux.envars --pid <pid>: HOSTNAME is the pod's name
Its command linelinux.psaux
Its filesystem viewlinux.mountinfo --pids <pid>, which shows the mount namespace
Its connectionslinux.sockstat --pids <pid>, with the network namespace
Code injected into memorylinux.malware.malfind
Files it held openlinux.lsof --pid <pid>

Volatility has no plugin for cgroups, so the pod name in the environment and the mount namespace are the ties from a process to a container.

Cautions

  • A node image is as large as the node's memory, and taking it takes minutes to tens of minutes. Every container on the node is in it.
  • Hash while you stream, and record the hash of the AVML binary you ran.
  • Writing a memory image or a checkpoint to the node's own disk changes the disk evidence. Stream it off, or snapshot the disk first.
  • Both a checkpoint and a node image contain secrets in clear: tokens, keys, passwords. Encrypt them and limit who can read them.

The container evidence template keeps the node and records the pod; these runbooks add memory to it.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.