Home/For responders/Clearing an attacker out of a cluster
Clearing an attacker out of a cluster
Deleting the pod an attacker used closes one way in. An attacker with the right permissions can leave others that outlive the pod, and some that no change to roles or bindings removes.
Credentials that outlive the pod
| Credential | Why it survives | How to end it |
|---|---|---|
| A client certificate | Kubernetes cannot revoke a certificate. Anyone allowed to create a signing request and approve it for kubernetes.io/kube-apiserver-client can issue one under any name, valid for up to a year by default. Approved requests are deleted after an hour, so only the audit log shows them | Rotate the cluster's certificate authority |
Membership of system:masters | It skips every authorization check, and no binding grants it, so none can take it away | Rotate the certificate authority |
A token made with kubectl create token | There is no record of tokens issued and no way to revoke one | Delete and recreate the service account, which ends every token it has |
| A pod's own token | It ends when the pod is deleted, for the API server. A system that checks the token offline keeps accepting it until it expires, and injected tokens can be extended to a year | Delete the pod. Systems that check it offline accept it until it expires |
| A long-lived token Secret | A Secret of type kubernetes.io/service-account-token never expires. The built-in cleanup only touches tokens Kubernetes made itself, after a year unused | Delete the Secret |
| Cloud credentials | The node's role and any workload identity were reachable from the pod | Revoke them in the cloud. The AWS, Azure, Google Cloud, and Oracle Cloud pages say how |
Where to look for persistence
| Place | What it gives an attacker | Look with |
|---|---|---|
| Role bindings | Rights for an identity they control | kubectl get clusterrolebindings,rolebindings -A -o wide, and creates in the audit log |
| Admission webhooks and mutating policies | A mutating webhook reads and can change every object it is registered for, such as adding a container to every new pod, your clean replacements included | kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations, and mutatingadmissionpolicies on 1.36 and later |
| DaemonSets and CronJobs | A pod on every node, or one that comes back on a schedule | kubectl get daemonsets,cronjobs -A |
| Token Secrets and service accounts | Credentials that do not expire | kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token |
| Ephemeral containers | A container added to a running pod, which cannot be removed | Pods whose spec lists ephemeralContainers. Delete the pod |
| Static pods | A pod the kubelet runs from a file on the node. Deleting it through the API does nothing, and one with an invalid namespace never appears there at all | The node's static pod directory, on a disk made from its snapshot |
Static pods, the kubelet's own API, etcd, and the container runtime socket are not seen by admission control or the audit log. An attacker with root on a node can work through them unrecorded, which is why the node is replaced.
Rotating the cluster certificate authority
| Service | How | What it costs |
|---|---|---|
| GKE | gcloud container clusters update <cluster> --start-credential-rotation, update every client, then --complete-credential-rotation. Also rotates the service account signing keys and the control plane IP | Nodes are recreated. Zonal Standard clusters have brief control plane downtime. Completes itself after seven days |
| EKS | aws eks create-certificate-authority, then activate-certificate-authority once nodes trust it, then delete-certificate-authority on the old one. Until that last step, the old authority stays trusted | You replace managed and self-managed nodes yourself, and Karpenter nodes only if drift detection is on. AWS handles the control plane, Fargate, and Auto Mode |
| AKS | az aks rotate-certs. Rotates every certificate and authority, and refreshes service account tokens | Every node and disk is recreated, with up to 30 minutes of downtime |
| OKE | oci ce cluster start-credential-rotation --cluster-id <ocid> --auto-completion-delay-duration P5D, new kubeconfigs for every client, restart the nodes or the pods that call the API server, then complete-credential-rotation | Old authorities are removed only when you complete it, or after the delay you set |
Until the old authority is removed, every certificate it signed still works. Start the rotation in the same window as the rest of the clean-up, and finish it as soon as clients and nodes have moved.
Replace the nodes
A node the attacker reached may have a static pod, a changed kubelet
configuration, or a tool you will not find. Drain it with
kubectl drain --ignore-daemonsets <node> and delete the machine,
once its disks are preserved. AWS's GuardDuty guidance says to terminate a
compromised node; Google's lists deleting the VM as the last resort.
Bring the workloads back from images you trust. A digest names exactly one image and a tag can be moved, so deploy by digest, and verify signatures at admission where you can. Remove the image the attacker used from your registry. The admission control template refuses risky pods before they start.
What to watch in the audit log
- Creates on
clusterrolebindings,rolebindings, andserviceaccounts/token - Any change to
mutatingwebhookconfigurations,validatingwebhookconfigurations, ormutatingadmissionpolicies - Creates on
certificatesigningrequests, and updates on theirapprovalsubresource - Requests on
pods/exec,pods/ephemeralcontainers, andnodes/proxy, whatever the verb - The annotation
authentication.k8s.io/legacy-token-invalidated, which marks a request made with a token Kubernetes already invalidated
The cluster audit log template keeps the log and alerts on exec and on new cluster role bindings.