Home/For responders/Kubernetes for defenders
Kubernetes for defenders
Most Kubernetes compromises follow a handful of paths. Each one leaves traces in a different place, so knowing the path tells you where to look.
Four layers, three kinds of evidence
| Layer | Where its evidence is |
|---|---|
| Container and node | Runtime detection that watches system calls, such as Falco, plus the node's own disk and memory |
| Cluster API | The Kubernetes audit log: every request, who made it, and what it touched |
| Cloud account | CloudTrail or Cloud Audit Logs, where the node's role and workload identities show up |
Attack paths and what they leave
| Path | What shows it | What stops it |
|---|---|---|
| Escape through a privileged pod or a host mount | Audit log: a pod created with privileged, hostPath, or hostPID. Runtime: nsenter or a mount from inside a container | An admission policy refusing those settings |
| A stolen service account token | Runtime: the token file read by a shell or curl. Audit log: that account calling from an unusual address | Not mounting tokens where they are not needed, and narrow roles |
| Exec into a running pod | Audit log: a request on the pods/exec subresource, with the user. kubectl 1.31 and later sends it as a get, older clients as a create, so match the subresource | Granting pods/exec to few people, and alerting on it in production |
| The node's cloud credentials | Cloud audit log: the node's role used from somewhere it should not be | On EKS, IMDSv2 with a hop limit of 1, which keeps pods off the metadata service |
| Borrowing a workload identity | Audit log: a pod created with a service account bound to a cloud role. Cloud audit log: that role assumed | Limiting who may create pods in that namespace, and trust conditions naming the exact service account |
| Bypassing admission control | Audit log: changes to webhook configurations, or pods created in an exempt namespace | Security policies set to fail closed, and alerts on any change to the webhooks |
| Escalating through role bindings | Audit log: a new cluster role binding, especially to cluster-admin | Few people with the bind and escalate verbs, and an alert on every new binding |
The workload identity row matters more than it looks. Anyone who can create a pod using a service account bound to a cloud role effectively has that role. Reviewing who can create pods is reviewing cloud access.
Keeping a compromised pod for investigation
Deleting the pod deletes the evidence. Keep it running and cut it off:
- Take it out of its controller. Remove the label its deployment selects on. The deployment starts a replacement, so the service keeps running, and leaves this pod alone.
- Cut its network. Add a quarantine label and a network policy that denies all traffic to and from pods with that label. This only works if the cluster's network plugin enforces network policies. Check that before an incident.
- Cordon the node, so nothing new is scheduled next to it.
- Capture. Save the pod's full definition and its logs, then snapshot the node's disks. The container evidence template does this on GKE, EKS, AKS, and OKE. Container forensics covers checkpointing a single container, and Containers on GKE and EKS and Containers on AKS and OKE what changes when you cannot reach the node. Once the evidence is safe, Clearing an attacker out of a cluster covers what outlives the pod.
Avoid exec into the pod if you can. It changes the evidence, and whoever compromised it may be watching.
Where the cluster logs are
EKS sends control plane logs, including the audit log, to CloudWatch Logs only when you switch them on, per log type.
GKE records administrative actions by default. Reads, such as someone listing secrets, are Data Access logs and are off by default.
The cluster audit logs template keeps both, turns five attack paths into alerts, and at Standard sends every entry to the SIEM.