iac.htora.dev · security templates

Home/Templates/Evidence from a compromised container

Evidence from a compromised container

One script keeps a pod's node alive, records the pod, snapshots the node's disks, and cuts the pod off, on GKE, EKS, AKS, or OKE.

Terraform

Why bother

A container's evidence is spread across three places and held by none of them for long. The writable layer and memory live on a node the autoscaler may remove. The logs rotate. The pod's definition disappears with the pod. This template gives responders the rights to capture all three and nothing more, and a bucket that will not let anyone delete what they collect. A script runs the steps in the right order and writes down each one.

How you know it worked

Run the capture against a test pod. Its record, logs, and hashes are in the bucket, the node's disks have snapshots labelled with the incident, the pod carries quarantine=true, and deleting evidence is refused.

How it works

1The responder names the podincident id, namespace, pod2Its node is keptout of scale-down, cordoned, protected from termination on EKS3The pod is recordeddefinition, images by digest, logs, events, node4The node's disks are snapshottedor backed up on OKE, persistent volumes included, fixed at that instant5The pod is cut offquarantine=true, which the quarantine network policy matches6Every file is hashed and storedfor evidence-verify to check later
Step by step, from the thing that happens to the result you want.

Set it up

Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.

Terraform. An Object Lock evidence bucket, a responder role with an EKS access entry, and a capture script for EKS

What you need first

  • An EKS cluster with the incident-responder cluster role applied, and an authentication mode that allows access entries.
  • The IAM principals your responders sign in as, such as an SSO role.
  • For the capture: the AWS CLI, kubectl, and sha256sum, such as in CloudShell.

What it creates

  • An evidence bucket with Object Lock, versioning, and public access blocked
  • A responder role that can protect a node from termination and scale-in, snapshot its volumes, and add evidence, and nothing more
  • An access entry that puts the role in the cluster's incident-responders group
  • The capture commands, printed with your names filled in
  • Your own key, a year of Object Lock in the mode you choose, one-hour sessions, TLS-only access, and capture.sh, which logs every step and keeps going past a failure (Standard)

The code

Quick startReadyone file, 215 lines
One file. A bucket locked for a day, the responder role and access entry, and the commands to run.
Change these first
  • cluster = "your-cluster" main.tf, line 25
  • responders = ["arn:aws:iam::111122223333:role/Responder"] main.tf, line 26
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. terraform init
    Downloads the provider. Run it once per folder.
  3. terraform plan
    Shows what will be created. Changes nothing. Read it.
  4. terraform apply
    Shows the plan again and waits for you to type yes.
  5. terraform destroy
    When you have finished. If it refuses, How to undo it below explains why.
Download main.tf
Checks

Formatting passedValid passedLint passed

Security scan: 38 checks passed, and seven rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/container-evidence/aws/t0
main.tf
Open main.tf as plain text
StandardReadyfour files, 373 lines
Adds your own key, a year of Object Lock in the mode you choose, one-hour sessions, TLS-only access, and capture.sh with a log of every step.
Change these first
  • cluster. Name of the EKS cluster the capture script works on. Its authentication mode must allow access entries (API or API_AND_CONFIG_MAP).
  • responders. IAM principals allowed to take the responder role, such as your incident response SSO role.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. # terraform.tfvars
    cluster = ""
    responders = ""
    Create this file beside the others and fill in your values. Terraform reads it on its own.
  3. terraform init
    Downloads the provider. Run it once per folder.
  4. terraform plan
    Shows what will be created. Changes nothing. Read it.
  5. terraform apply
    Shows the plan again and waits for you to type yes.
  6. terraform output verify
    Prints the commands that prove it worked.
Download all four files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 65 checks passed, and seven rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/container-evidence/aws/t1
main.tf
Open main.tf as plain text
outputs.tf
Open outputs.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text
HardenedReadysix files, 238 lines
Standard as a module. Plan rules refuse the bucket or key deleted, Object Lock off or under 90 days, versioning off, public access, encryption other than KMS, and key rotation off. They also refuse a responder that can read or delete evidence or stop instances, and extra cluster groups or access policies. A daily plan reports drift, and the scan fails on any unexplained finding.
Change these first
  • cluster. Name of the EKS cluster the capture script works on. Its authentication mode must allow access entries (API or API_AND_CONFIG_MAP).
  • responders. IAM principals allowed to take the responder role, such as your incident response SSO role.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. # terraform.tfvars
    cluster = ""
    responders = ""
    Create this file beside the others and fill in your values. Terraform reads it on its own.
  3. terraform init
    Downloads the provider. Run it once per folder.
  4. terraform plan
    Shows what will be created. Changes nothing. Read it.
  5. terraform apply
    Shows the plan again and waits for you to type yes.
  6. terraform output verify
    Prints the commands that prove it worked.
Download all six files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 56 checks passed, and none flagged.

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/container-evidence/aws/t2
.checkov.yml
Open .checkov.yml as plain text
guard.yml
Open guard.yml as plain text
main.tf
Open main.tf as plain text
outputs.tf
Open outputs.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text

How to check it worked

Run the printed commands, or capture.sh at Standard, against a test pod. aws s3 ls shows its files with .sha256 beside each. Listing snapshots with the incident tag shows the node's volumes, and termination protection is on.

What catches people

On Fargate there is no node to keep or snapshot, and the pod's storage is deleted when it stops. The script records the pod and its logs and says so.

EKS Auto Mode nodes cannot be reached over SSH or SSM and are replaced after at most 21 days. Their snapshots have not been tested here.

Termination protection does not stop an Auto Scaling group removing an instance. Scale-in protection does, and the script sets both. Spot instances cannot take termination protection, so that step fails on them.

Karpenter's do-not-disrupt annotation stops its voluntary disruption only. Expiry, Spot interruption, node repair, and a manual delete still remove the node, so snapshot early.

The role has ec2:ModifyInstanceAttribute, which changes more than termination protection. Keep the role for responders and its sessions short.

The quarantine label only cuts traffic if the cluster enforces network policies. Check that before an incident.

How to undo it

terraform destroy removes the role and the access entry. The bucket cannot be deleted while it contains locked objects.

What it costs

EBS snapshot storage by the GB, S3 storage, and one KMS key at Standard.

Good to know

  • The Kubernetes rights come from the incident-responder cluster role in the Kubernetes evidence template: read, label pods, cordon nodes, and no exec, secrets, or deletes. Apply it to the cluster first.
  • Nothing here uses exec or runs a debugging pod on the node. Both change the evidence, so do them afterwards, knowingly, with the steps on the Containers on GKE and EKS, and Containers on AKS and OKE pages.
  • Memory is not captured. On EKS, the node is an EC2 instance, so tagging it for the evidence capture template adds a memory image, and isolates the whole node with it.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.