Home/For responders/Containers on GKE and EKS
Containers on GKE and EKS
The container is the same on both clouds. What differs is who owns the node, what the cluster records by default, and how a pod's identity shows up in the cloud's own logs.
Side by side
| GKE | EKS | |
|---|---|---|
| Cluster audit log | Changes are logged by default and cannot be switched off. Reads (get, list) are Data Access logs, off by default | Every control plane log is off by default. Switch on audit and the others one by one. They go to CloudWatch Logs in /aws/eks/<cluster>/cluster, best effort |
kubectl exec | Admin Activity, with protoPayload.methodName matching io.k8s.core.v1.pods.exec | The audit log, on the exec subresource: a get from kubectl 1.31 and later, a create from older clients |
| A pod's cloud identity | Workload Identity: the pod acts as a Google service account, or is granted access directly as its Kubernetes service account. The Cloud Audit Logs name whichever it used | IRSA: an STS AssumeRoleWithWebIdentity. Pod Identity: AssumeRoleForPodIdentity from eks-auth.amazonaws.com, with session tags naming the namespace, service account, pod, and pod UID |
| Node operating system | Container-Optimized OS: read-only root, toolbox for a Debian shell with the host at /media/root, crictl on containerd nodes | Amazon Linux or Bottlerocket. Bottlerocket has a read-only root; its admin container is off by default, and sheltie inside it gives a root shell on the host |
| What removes a node | The cluster autoscaler, and auto-repair after about 10 minutes NotReady | The Auto Scaling group, the cluster autoscaler, or Karpenter |
| Keeping a node | The annotation cluster-autoscaler.kubernetes.io/scale-down-disabled=true, and a cordon | Termination protection, scale-in protection, the same autoscaler annotation, and a cordon. Karpenter's karpenter.sh/do-not-disrupt stops its voluntary disruption only: expiry, Spot interruption, repair, and a manual delete still remove the node. Spot instances cannot take termination protection |
| Managed runtime detection | Container Threat Detection, in Security Command Center Premium or Enterprise | GuardDuty EKS Protection reads the audit log from its own stream. Runtime Monitoring adds an agent that sees processes, files, and connections |
Responding on GKE Standard
- Record the pod and its logs, and annotate its node so the autoscaler keeps it.
- Cordon the node. Auto-repair still replaces a node that stays NotReady, so move to the snapshot quickly.
- Snapshot the node's disks. A GKE node is a Compute Engine VM with the node's
name, so
gcloud compute instances describe <node>lists them, persistent volumes included. - Label the pod for quarantine, then investigate from a
toolboxon the node or a disk made from the snapshot.
Turn on Data Access logs for GKE before an incident. Without them, someone reading every secret in the cluster leaves nothing in the audit log. The cluster audit log template does it.
Responding on EKS with EC2 nodes
AWS's EKS best practices guide lists these steps, in this order:
- Identify the offending pod and worker node.
- Isolate the pod with a network policy that denies all ingress and egress.
- Revoke temporary credentials assigned to the pod or node, if necessary.
- Cordon the worker node.
- Enable termination protection on the node.
- Label the pod and node as part of an active investigation.
- Capture volatile artefacts on the node: memory, processes and open ports,
container state with
crictl, then snapshot the EBS volumes.
Termination protection does not stop an Auto Scaling group removing the instance. Set scale-in protection too. The node is an ordinary EC2 instance, so the evidence template can capture its memory when you tag it, at the cost of isolating every pod on it. Checkpoints and node memory has the runbooks for capturing memory without isolating the node, on EKS and GKE.
When you cannot reach the node
| Mode | What you lose | What to rely on |
|---|---|---|
| GKE Autopilot | SSH to nodes, privileged pods, host namespaces. A node cordoned for about 10 minutes is repaired, which replaces it | The pod's record and logs, the audit logs, Container Threat Detection findings. Do not cordon |
| EKS on Fargate | The node entirely: no SSH, no DaemonSets, no privileged pods. The pod's storage is deleted when it stops. GuardDuty Runtime Monitoring does not cover EKS on Fargate | The pod's record and logs, the control plane logs, CloudTrail, and logs the application ships. Isolate with a deny-all network policy |
| EKS Auto Mode | SSH and SSM. Nodes run Bottlerocket and are replaced after at most 21 days | kubectl debug node/<node> --profile=sysadmin, aws ec2 get-console-output, and a NodeDiagnostic resource that uploads the node's logs to S3 |
In all three, the evidence you will have is the evidence you shipped off the node before the incident. Send container logs to a central store, keep the cluster audit log on, and run managed runtime detection where it is supported.
Templates for this
- Evidence from a compromised container, on GKE and EKS
- Evidence that survives termination: quarantine policy and responder role for any cluster
- Cluster audit logs, kept and alerted on
- Runtime detection with Falco
- Workload identity, so a pod's cloud access is narrow and traceable