iac.htora.dev · security templates

Home/For responders/Clearing an attacker out of a cluster

Clearing an attacker out of a cluster

Deleting the pod an attacker used closes one way in. An attacker with the right permissions can leave others that outlive the pod, and some that no change to roles or bindings removes.

1Preservethe evidence first, before anything is deleted2Find every way back inthe audit log from before the first sign, and the cluster as it is now3Close them togetherbindings, webhooks, workloads, tokens, in one change window4Rotate what cannot be revokedthe cluster's certificate authority, and the cloud credentials5Replace the nodesnew machines from known images, never cleaned old ones6Watch for the returnthe same identities, addresses, and images
Eradication on Kubernetes. Closing one path at a time warns the attacker, who then uses another.

Credentials that outlive the pod

CredentialWhy it survivesHow to end it
A client certificateKubernetes cannot revoke a certificate. Anyone allowed to create a signing request and approve it for kubernetes.io/kube-apiserver-client can issue one under any name, valid for up to a year by default. Approved requests are deleted after an hour, so only the audit log shows themRotate the cluster's certificate authority
Membership of system:mastersIt skips every authorization check, and no binding grants it, so none can take it awayRotate the certificate authority
A token made with kubectl create tokenThere is no record of tokens issued and no way to revoke oneDelete and recreate the service account, which ends every token it has
A pod's own tokenIt ends when the pod is deleted, for the API server. A system that checks the token offline keeps accepting it until it expires, and injected tokens can be extended to a yearDelete the pod. Systems that check it offline accept it until it expires
A long-lived token SecretA Secret of type kubernetes.io/service-account-token never expires. The built-in cleanup only touches tokens Kubernetes made itself, after a year unusedDelete the Secret
Cloud credentialsThe node's role and any workload identity were reachable from the podRevoke them in the cloud. The AWS, Azure, Google Cloud, and Oracle Cloud pages say how

Where to look for persistence

PlaceWhat it gives an attackerLook with
Role bindingsRights for an identity they controlkubectl get clusterrolebindings,rolebindings -A -o wide, and creates in the audit log
Admission webhooks and mutating policiesA mutating webhook reads and can change every object it is registered for, such as adding a container to every new pod, your clean replacements includedkubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations, and mutatingadmissionpolicies on 1.36 and later
DaemonSets and CronJobsA pod on every node, or one that comes back on a schedulekubectl get daemonsets,cronjobs -A
Token Secrets and service accountsCredentials that do not expirekubectl get secrets -A --field-selector type=kubernetes.io/service-account-token
Ephemeral containersA container added to a running pod, which cannot be removedPods whose spec lists ephemeralContainers. Delete the pod
Static podsA pod the kubelet runs from a file on the node. Deleting it through the API does nothing, and one with an invalid namespace never appears there at allThe node's static pod directory, on a disk made from its snapshot

Static pods, the kubelet's own API, etcd, and the container runtime socket are not seen by admission control or the audit log. An attacker with root on a node can work through them unrecorded, which is why the node is replaced.

Rotating the cluster certificate authority

ServiceHowWhat it costs
GKEgcloud container clusters update <cluster> --start-credential-rotation, update every client, then --complete-credential-rotation. Also rotates the service account signing keys and the control plane IPNodes are recreated. Zonal Standard clusters have brief control plane downtime. Completes itself after seven days
EKSaws eks create-certificate-authority, then activate-certificate-authority once nodes trust it, then delete-certificate-authority on the old one. Until that last step, the old authority stays trustedYou replace managed and self-managed nodes yourself, and Karpenter nodes only if drift detection is on. AWS handles the control plane, Fargate, and Auto Mode
AKSaz aks rotate-certs. Rotates every certificate and authority, and refreshes service account tokensEvery node and disk is recreated, with up to 30 minutes of downtime
OKEoci ce cluster start-credential-rotation --cluster-id <ocid> --auto-completion-delay-duration P5D, new kubeconfigs for every client, restart the nodes or the pods that call the API server, then complete-credential-rotationOld authorities are removed only when you complete it, or after the delay you set

Until the old authority is removed, every certificate it signed still works. Start the rotation in the same window as the rest of the clean-up, and finish it as soon as clients and nodes have moved.

Replace the nodes

A node the attacker reached may have a static pod, a changed kubelet configuration, or a tool you will not find. Drain it with kubectl drain --ignore-daemonsets <node> and delete the machine, once its disks are preserved. AWS's GuardDuty guidance says to terminate a compromised node; Google's lists deleting the VM as the last resort.

Bring the workloads back from images you trust. A digest names exactly one image and a tag can be moved, so deploy by digest, and verify signatures at admission where you can. Remove the image the attacker used from your registry. The admission control template refuses risky pods before they start.

What to watch in the audit log

  • Creates on clusterrolebindings, rolebindings, and serviceaccounts/token
  • Any change to mutatingwebhookconfigurations, validatingwebhookconfigurations, or mutatingadmissionpolicies
  • Creates on certificatesigningrequests, and updates on their approval subresource
  • Requests on pods/exec, pods/ephemeralcontainers, and nodes/proxy, whatever the verb
  • The annotation authentication.k8s.io/legacy-token-invalidated, which marks a request made with a token Kubernetes already invalidated

The cluster audit log template keeps the log and alerts on exec and on new cluster role bindings.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.