Home/Templates/Evidence from a compromised container
Evidence from a compromised container
One script keeps a pod's node alive, records the pod, snapshots the node's disks, and cuts the pod off, on GKE, EKS, AKS, or OKE.
Terraform
Why bother
A container's evidence is spread across three places and held by none of them for long. The writable layer and memory live on a node the autoscaler may remove. The logs rotate. The pod's definition disappears with the pod. This template gives responders the rights to capture all three and nothing more, and a bucket that will not let anyone delete what they collect. A script runs the steps in the right order and writes down each one.
Run the capture against a test pod. Its record, logs, and hashes are in the bucket, the node's disks have snapshots labelled with the incident, the pod carries quarantine=true, and deleting evidence is refused.
How it works
Set it up
Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.
Terraform. An Object Lock evidence bucket, a responder role with an EKS access entry, and a capture script for EKS
What you need first
- An EKS cluster with the incident-responder cluster role applied, and an authentication mode that allows access entries.
- The IAM principals your responders sign in as, such as an SSO role.
- For the capture: the AWS CLI, kubectl, and sha256sum, such as in CloudShell.
What it creates
- An evidence bucket with Object Lock, versioning, and public access blocked
- A responder role that can protect a node from termination and scale-in, snapshot its volumes, and add evidence, and nothing more
- An access entry that puts the role in the cluster's incident-responders group
- The capture commands, printed with your names filled in
- Your own key, a year of Object Lock in the mode you choose, one-hour sessions, TLS-only access, and capture.sh, which logs every step and keeps going past a failure (Standard)
The code
Quick startReady
cluster = "your-cluster"main.tf, line 25responders = ["arn:aws:iam::111122223333:role/Responder"]main.tf, line 26
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: 38 checks passed, and seven rules flagged, each with a written reason:
- S3 lifecycle configuration
- S3 event notifications
- IAM write access without constraints
- S3 cross-region replication
- S3 encrypted with KMS
- S3 server access logging
- IAM resource of *
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open main.tf as plain text
StandardReady
cluster. Name of the EKS cluster the capture script works on. Its authentication mode must allow access entries (API or API_AND_CONFIG_MAP).responders. IAM principals allowed to take the responder role, such as your incident response SSO role.
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.# terraform.tfvars cluster = "" responders = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 65 checks passed, and seven rules flagged, each with a written reason:
- S3 lifecycle configuration
- S3 event notifications
- IAM permissions management without constraints
- IAM write access without constraints
- S3 cross-region replication
- S3 server access logging
- IAM resource of *
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
cluster. Name of the EKS cluster the capture script works on. Its authentication mode must allow access entries (API or API_AND_CONFIG_MAP).responders. IAM principals allowed to take the responder role, such as your incident response SSO role.
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.# terraform.tfvars cluster = "" responders = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 56 checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
Run the printed commands, or capture.sh at Standard, against a test pod. aws s3 ls shows its files with .sha256 beside each. Listing snapshots with the incident tag shows the node's volumes, and termination protection is on.
On Fargate there is no node to keep or snapshot, and the pod's storage is deleted when it stops. The script records the pod and its logs and says so.
EKS Auto Mode nodes cannot be reached over SSH or SSM and are replaced after at most 21 days. Their snapshots have not been tested here.
Termination protection does not stop an Auto Scaling group removing an instance. Scale-in protection does, and the script sets both. Spot instances cannot take termination protection, so that step fails on them.
Karpenter's do-not-disrupt annotation stops its voluntary disruption only. Expiry, Spot interruption, node repair, and a manual delete still remove the node, so snapshot early.
The role has ec2:ModifyInstanceAttribute, which changes more than termination protection. Keep the role for responders and its sessions short.
The quarantine label only cuts traffic if the cluster enforces network policies. Check that before an incident.
How to undo it
terraform destroy removes the role and the access entry. The bucket cannot be deleted while it contains locked objects.
What it costs
EBS snapshot storage by the GB, S3 storage, and one KMS key at Standard.
Terraform. An immutable blob container, a node reader role, a snapshot-and-write responder role, and a capture script for AKS
What you need first
- An AKS cluster with Microsoft Entra ID integration, and the incident-responder cluster role bound to your responder group by its object ID.
- A Microsoft Entra group for responders. Whoever applies this needs Owner, or Contributor plus User Access Administrator, on the subscription.
- For the capture: the Azure CLI, kubectl, kubelogin, and sha256sum, such as in Cloud Shell.
What it creates
- A resource group, outside the node resource group, for the snapshots and the evidence account
- A storage account with shared keys off and a blob container under a time-based immutability policy
- A role that reads node VMs and their disks in the node resource group
- A role that creates snapshots and adds blobs in the evidence group, with no read or delete of evidence
- Azure Kubernetes Service Cluster User Role on the cluster, enough to fetch a kubeconfig
- Your own key from Key Vault, a year of retention you can lock, several responder groups, and optional logging of every access to the evidence (Standard)
- capture.sh, which logs every step and keeps going past a failure (Standard)
The code
Quick startReady
cluster_name = "your-cluster"main.tf, line 30cluster_resource_group = "your-cluster-rg"main.tf, line 31responder_group_object_id = "00000000-0000-0000-0000-000000000000"main.tf, line 32
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: 12 checks passed, and seven rules flagged, each with a written reason:
- Storage encrypted with your own key
- Classic blob read logging
- Storage private endpoint
- Storage soft delete
- Storage replication
- Classic queue logging
- Storage public network access off
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open main.tf as plain text
StandardReady
subscription_id. Subscription that contains the cluster. Snapshots of a disk must be in the disk's own subscription, so the evidence goes here too.cluster_name. Name of the AKS cluster the capture script works on. It must use Microsoft Entra ID integration.cluster_resource_group. The cluster's own resource group. Its MC_ node resource group is looked up from the cluster.responder_group_object_ids. Object IDs of the Microsoft Entra groups allowed to capture. Bind the same IDs to the incident-responder cluster role.
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.# terraform.tfvars subscription_id = "" cluster_name = "" cluster_resource_group = "" responder_group_object_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 18 checks passed, and 10 rules flagged, each with a written reason:
- Classic blob read logging
- Key Vault private endpoint
- Storage private endpoint
- Key Vault firewall
- Key backed by an HSM
- Key Vault public network access off
- Classic queue logging
- Storage firewall denies by default
- Key expiration date
- Storage public network access off
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open keys.tf as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
subscription_id. Subscription that contains the cluster. Snapshots of a disk must be in the disk's own subscription, so the evidence goes here too.cluster_name. Name of the AKS cluster the capture script works on. It must use Microsoft Entra ID integration.cluster_resource_group. The cluster's own resource group. Its MC_ node resource group is looked up from the cluster.responder_group_object_ids. Object IDs of the Microsoft Entra groups allowed to capture. Bind the same IDs to the incident-responder cluster role.
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.# terraform.tfvars subscription_id = "" cluster_name = "" cluster_resource_group = "" responder_group_object_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 18 checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
Run the printed commands, or capture.sh at Standard, against a test pod. Snapshots of the node's disks appear in the evidence group, tagged with the incident, and the pod carries quarantine=true. Someone who can delete blobs, such as Storage Blob Data Owner, is still refused.
AKS puts the OS disk on the host's own storage by default when the VM size allows it. That ephemeral disk cannot be snapshotted, and a reimage or removal erases it. The script records whether the OS disk is ephemeral and snapshots every managed disk, persistent volumes included.
AKS does not support changes to anything in the node resource group, so nothing here protects the scale set instance. The annotation keeps the cluster autoscaler away, and do-not-disrupt keeps node auto-provisioning from consolidating the node. A manual delete or a node image upgrade still removes it, so snapshot early.
Node auto-repair reboots, then reimages, a node that stops reporting Ready for about 10 minutes. Cutting the node off the network can trigger that. Cordoning does not.
On virtual nodes, pods run in Azure Container Instances. There is no VM or disk to keep, and the script records the pod and its logs and says so.
Clusters with node resource group lockdown, including AKS Automatic, add a deny assignment on the node group. Snapshotting their disks into another group has not been tested here.
The node reader role has disks/beginGetAccess/action, as Microsoft's own Disk Backup Reader role does. It also lets a responder create a download link for a disk in the node group. Remove it if snapshots work without it in your tenant.
The quarantine label only cuts traffic if the cluster enforces network policies. Check that before an incident.
How to undo it
terraform destroy removes the roles and assignments. The container cannot be deleted while it contains evidence under its policy, and a locked policy can only be extended. Snapshots stay until you delete them.
What it costs
Incremental snapshot storage by the GB used, blob storage, and one Key Vault key at Standard.
Terraform. A retention-locked evidence bucket, a snapshot-only custom role, and a capture script for GKE
What you need first
- A GKE cluster with the incident-responder cluster role applied and bound, through Google Groups for RBAC, to your responder group's email address. The group must be a member of gke-security-groups@ in your domain.
- A Google group for responders, and Project IAM Admin and Storage Admin for whoever runs apply.
- For the capture: gcloud, kubectl, and sha256sum, such as in Cloud Shell.
What it creates
- An evidence bucket with uniform access, public access prevention, and a retention period
- Rights for responders to add evidence and list object names, but not read, change, or delete evidence
- A custom role that can read a node VM and snapshot its disks, and nothing more, plus Cluster Viewer to fetch credentials
- The capture commands, printed with your names filled in
- Your own key on the bucket, a retention period you choose and can lock, and capture.sh, which logs every step and keeps going past a failure (Standard)
The code
Quick startReady
project_id = "your-project-id"main.tf, line 24cluster = "your-cluster"main.tf, line 25cluster_location = "us-central1"main.tf, line 26responders = "group:incident-responders@example.com"main.tf, line 27
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: 13 checks passed, and two rules flagged, each with a written reason:
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open main.tf as plain text
StandardReady
project_id. Project that contains the cluster, its node VMs, and the evidence bucket.cluster. Name of the GKE cluster the capture script works on.cluster_location. Region or zone of the cluster.responders. Who may capture, as IAM members such as group:incident-responders@example.com. Bind the same group, by its email address, to the incident-responder cluster role through Google Groups for RBAC.
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.# terraform.tfvars project_id = "" cluster = "" cluster_location = "" responders = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 12 checks passed, and two rules flagged, each with a written reason:
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
project_id. Project that contains the cluster, its node VMs, and the evidence bucket.cluster. Name of the GKE cluster the capture script works on.cluster_location. Region or zone of the cluster.responders. Who may capture, as IAM members such as group:incident-responders@example.com. Bind the same group, by its email address, to the incident-responder cluster role through Google Groups for RBAC.
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.# terraform.tfvars project_id = "" cluster = "" cluster_location = "" responders = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 12 checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
Run the printed commands, or capture.sh at Standard, against a test pod. gcloud storage ls shows its files with .sha256 beside each. Listing snapshots with the incident label shows the node's disks, and removing an object is refused.
On Autopilot, a node cordoned for about 10 minutes is repaired, which replaces it, and you cannot reach nodes over SSH. The Standard script skips the cordon there and records whether the node's disks could be snapshotted.
Node auto-repair also replaces a node that stays NotReady for about 10 minutes. Snapshot early: the snapshot survives the node.
A cordon stops new pods landing on the node. It does not stop the pod's own controller from replacing it, so take the pod out of its deployment's selector if you need it kept.
The quarantine label only cuts traffic if the cluster enforces network policies. Check that before an incident.
How to undo it
terraform destroy removes the role and the grants. The bucket cannot be deleted while it contains evidence under retention, and a locked retention cannot be shortened.
What it costs
Snapshot storage by the GB, and bucket storage. The roles cost nothing.
Terraform. A retention-ruled bucket, IAM policies for backups and writes, and a capture script for OKE
What you need first
- An OKE cluster with managed nodes, and the incident-responder cluster role bound to your responder group by its OCID.
- An IAM group for responders, and rights to create policies in the cluster's, the nodes', and the evidence compartments for whoever applies this.
- For the capture: the OCI CLI, kubectl, and sha256sum, such as in Cloud Shell.
What it creates
- An evidence bucket with a time-based retention rule and no public access
- Policies that let responders fetch a kubeconfig, read nodes, and back up their boot and block volumes, with no deletes
- A policy that lets responders add objects and list their names. It grants no read, overwrite, or delete
- The capture commands, printed with your names filled in
- Your own key from Vault, a year of retention with a lock you can schedule, separate compartments for nodes and evidence, and several groups (Standard)
- capture.sh, which logs every step and keeps going past a failure (Standard)
The code
Quick startReady
region = "us-ashburn-1"main.tf, line 29cluster_id = "ocid1.cluster.oc1..replace-me"main.tf, line 30responder_group_ocid = "ocid1.group.oc1..replace-me"main.tf, line 31
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: two checks passed, and three rules flagged, each with a written reason:
Last run 2026-10-02 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open main.tf as plain text
StandardReady
region. Region identifier of the cluster, such as us-ashburn-1. The bucket and its key go here too.cluster_id. OCID of the OKE cluster the capture script works on.responder_group_ocids. OCIDs of the IAM groups allowed to capture. Bind the same OCIDs to the incident-responder cluster role.
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.# terraform.tfvars region = "" cluster_id = "" responder_group_ocids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: three checks passed, and two rules flagged, each with a written reason:
Last run 2026-10-02 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open keys.tf as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
region. Region identifier of the cluster, such as us-ashburn-1. The bucket and its key go here too.cluster_id. OCID of the OKE cluster the capture script works on.responder_group_ocids. OCIDs of the IAM groups allowed to capture. Bind the same OCIDs to the incident-responder cluster role.
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.# terraform.tfvars region = "" cluster_id = "" responder_group_ocids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: three checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
Run the printed commands, or capture.sh at Standard, against a test pod. Its files appear in the bucket with .sha256 beside each, and the node's volume backups carry the incident tag. Someone who can delete objects in the bucket is still refused.
OCI has no termination protection for an instance. Anyone who can manage the node pool can cycle or delete the node, so ask the platform team to pause node pool changes until the backups exist.
Replacing a node's boot volume, by node cycling or by hand, swaps the disk in place. Back it up first.
Virtual nodes run in Oracle's tenancy, with no instance or volume of yours. The script records the pod and its logs and says so. Exec is not available there at all.
Persistent volumes from the oci-bv storage class are deleted with their claim by default. The backups keep their contents; set the volume's reclaim policy to Retain if the claim may be deleted before the investigation ends.
Uploads use --no-multipart, because multipart uploads need read and overwrite rights that responders do not have. That limits a single file to what one PutObject call accepts.
Responders cannot delete backups, but anyone with manage boot-volume-backups can. Block Volume offers a retention lock for backups, which this template does not set.
The quarantine label only cuts traffic if the cluster enforces network policies. Check that before an incident.
How to undo it
terraform destroy removes the policies. The bucket cannot be deleted while objects are under retention, and once the rule locks, only an empty bucket can be deleted. Backups stay until you delete them.
What it costs
Volume backup storage by the GB, Object Storage, and one Vault key at Standard. Full backups cost more than incremental ones and stand alone.
Good to know
- The Kubernetes rights come from the incident-responder cluster role in the Kubernetes evidence template: read, label pods, cordon nodes, and no exec, secrets, or deletes. Apply it to the cluster first.
- Nothing here uses exec or runs a debugging pod on the node. Both change the evidence, so do them afterwards, knowingly, with the steps on the Containers on GKE and EKS, and Containers on AKS and OKE pages.
- Memory is not captured. On EKS, the node is an EC2 instance, so tagging it for the evidence capture template adds a memory image, and isolates the whole node with it.