iac.htora.dev · security templates

Home/For responders/Containers on GKE and EKS

Containers on GKE and EKS

The container is the same on both clouds. What differs is who owns the node, what the cluster records by default, and how a pod's identity shows up in the cloud's own logs.

Side by side

GKEEKS
Cluster audit logChanges are logged by default and cannot be switched off. Reads (get, list) are Data Access logs, off by defaultEvery control plane log is off by default. Switch on audit and the others one by one. They go to CloudWatch Logs in /aws/eks/<cluster>/cluster, best effort
kubectl execAdmin Activity, with protoPayload.methodName matching io.k8s.core.v1.pods.execThe audit log, on the exec subresource: a get from kubectl 1.31 and later, a create from older clients
A pod's cloud identityWorkload Identity: the pod acts as a Google service account, or is granted access directly as its Kubernetes service account. The Cloud Audit Logs name whichever it usedIRSA: an STS AssumeRoleWithWebIdentity. Pod Identity: AssumeRoleForPodIdentity from eks-auth.amazonaws.com, with session tags naming the namespace, service account, pod, and pod UID
Node operating systemContainer-Optimized OS: read-only root, toolbox for a Debian shell with the host at /media/root, crictl on containerd nodesAmazon Linux or Bottlerocket. Bottlerocket has a read-only root; its admin container is off by default, and sheltie inside it gives a root shell on the host
What removes a nodeThe cluster autoscaler, and auto-repair after about 10 minutes NotReadyThe Auto Scaling group, the cluster autoscaler, or Karpenter
Keeping a nodeThe annotation cluster-autoscaler.kubernetes.io/scale-down-disabled=true, and a cordonTermination protection, scale-in protection, the same autoscaler annotation, and a cordon. Karpenter's karpenter.sh/do-not-disrupt stops its voluntary disruption only: expiry, Spot interruption, repair, and a manual delete still remove the node. Spot instances cannot take termination protection
Managed runtime detectionContainer Threat Detection, in Security Command Center Premium or EnterpriseGuardDuty EKS Protection reads the audit log from its own stream. Runtime Monitoring adds an agent that sees processes, files, and connections

Responding on GKE Standard

  1. Record the pod and its logs, and annotate its node so the autoscaler keeps it.
  2. Cordon the node. Auto-repair still replaces a node that stays NotReady, so move to the snapshot quickly.
  3. Snapshot the node's disks. A GKE node is a Compute Engine VM with the node's name, so gcloud compute instances describe <node> lists them, persistent volumes included.
  4. Label the pod for quarantine, then investigate from a toolbox on the node or a disk made from the snapshot.

Turn on Data Access logs for GKE before an incident. Without them, someone reading every secret in the cluster leaves nothing in the audit log. The cluster audit log template does it.

Responding on EKS with EC2 nodes

AWS's EKS best practices guide lists these steps, in this order:

  1. Identify the offending pod and worker node.
  2. Isolate the pod with a network policy that denies all ingress and egress.
  3. Revoke temporary credentials assigned to the pod or node, if necessary.
  4. Cordon the worker node.
  5. Enable termination protection on the node.
  6. Label the pod and node as part of an active investigation.
  7. Capture volatile artefacts on the node: memory, processes and open ports, container state with crictl, then snapshot the EBS volumes.

Termination protection does not stop an Auto Scaling group removing the instance. Set scale-in protection too. The node is an ordinary EC2 instance, so the evidence template can capture its memory when you tag it, at the cost of isolating every pod on it. Checkpoints and node memory has the runbooks for capturing memory without isolating the node, on EKS and GKE.

When you cannot reach the node

ModeWhat you loseWhat to rely on
GKE AutopilotSSH to nodes, privileged pods, host namespaces. A node cordoned for about 10 minutes is repaired, which replaces itThe pod's record and logs, the audit logs, Container Threat Detection findings. Do not cordon
EKS on FargateThe node entirely: no SSH, no DaemonSets, no privileged pods. The pod's storage is deleted when it stops. GuardDuty Runtime Monitoring does not cover EKS on FargateThe pod's record and logs, the control plane logs, CloudTrail, and logs the application ships. Isolate with a deny-all network policy
EKS Auto ModeSSH and SSM. Nodes run Bottlerocket and are replaced after at most 21 dayskubectl debug node/<node> --profile=sysadmin, aws ec2 get-console-output, and a NodeDiagnostic resource that uploads the node's logs to S3

In all three, the evidence you will have is the evidence you shipped off the node before the incident. Send container logs to a central store, keep the cluster audit log on, and run managed runtime detection where it is supported.

Templates for this

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.