iac.htora.dev · security templates

Home/Templates/Evidence that survives termination

Evidence that survives termination

One tag or label keeps a machine alive, captures its disks and, on AWS, its memory, and cuts it off, in that order, on four clouds and Kubernetes.

TerraformArgo CD

Why bother

Cloud machines are built to disappear. Autoscaling replaces unhealthy instances, and Kubernetes reschedules pods. The most common first move in an incident, terminating the bad machine, destroys the disk, the memory, and the details the investigation needs. This template turns preservation into one action anyone on the incident can take: add a tag. Everything else happens in the right order, the same way every time, including at three in the morning.

How you know it worked

Run the drill on a test machine: tag or label it incident=drill-1. Its disks are captured, it is cut off from the network, and deleting it is refused. Each platform's tab gives the exact commands.

How it works

1Someone tags or labels the machineincident=<id>, from the console, the CLI, or a playbook2It is kept aliveprotected from deletion, and out of its autoscaling group3Its disks are capturedsnapshots at one instant, then memory where the cloud allows4It is cut offa quarantine group, a firewall tag, or a network policy
Step by step, from the thing that happens to the result you want.

Set it up

Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.

Terraform. EventBridge and a Lambda responder, with memory captured by AVML through Systems Manager

What you need first

  • An account where you can create Lambda functions, EventBridge rules, and IAM roles.
  • For memory capture: Linux instances managed by Systems Manager, with the AWS CLI, and a network path to S3 such as a gateway endpoint.
  • The AVML binary staged in the evidence bucket before an incident. The stage_avml output prints the command.

What it creates

  • A responder function, and a rule that fires when an instance is tagged incident=<id>
  • A quarantine security group per VPC, created on first use, with no inbound or outbound rules
  • A locked, encrypted evidence bucket containing each instance's metadata, its memory image, and a hash (Standard)
  • A Systems Manager document that captures memory with AVML, and a rule that isolates the instance once capture finishes (Standard)
  • A status topic for the incident channel, and a queue that keeps any event the responder failed on (Standard)

See the two alerts this creates

The code

Quick startReadyone file, 216 lines
One file, code inline. Tag-triggered responder that protects, detaches, snapshots, and isolates.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. terraform init
    Downloads the provider. Run it once per folder.
  3. terraform plan
    Shows what will be created. Changes nothing. Read it.
  4. terraform apply
    Shows the plan again and waits for you to type yes.
  5. terraform output verify
    Prints the commands that prove it worked.
  6. terraform destroy
    When you have finished. If it refuses, How to undo it below explains why.
Download main.tf
Checks

Formatting passedValid passedLint passed

Security scan: 35 checks passed, and eight rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/archive 2.8.1, hashicorp/aws 6.66.0.

outcomes/evidence-capture/aws/t0
main.tf
Open main.tf as plain text
StandardReadysix files, 696 lines
Adds the instance's metadata recorded before anything changes, memory capture with AVML, and a locked evidence bucket with its own key. It also adds isolation that waits for capture, status messages, and a queue for failed events.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. terraform init
    Downloads the provider. Run it once per folder.
  3. terraform plan
    Shows what will be created. Changes nothing. Read it.
  4. terraform apply
    Shows the plan again and waits for you to type yes.
  5. terraform output verify
    Prints the commands that prove it worked.
Download all six files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 80 checks passed, and 11 rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/archive 2.8.1, hashicorp/aws 6.66.0.

outcomes/evidence-capture/aws/t1
evidence.tf
Open evidence.tf as plain text
outputs.tf
Open outputs.tf as plain text
responder.py
Open responder.py as plain text
responder.tf
Open responder.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text
HardenedReadysix files, 246 lines
Standard as a module. Plan rules refuse the evidence bucket or key deleted, Object Lock or default retention removed, versioning off, and public access allowed. They also refuse encryption other than KMS, unpatterned capture parameters, and the responder losing its dead-letter queue. A daily plan reports drift, and the scan fails on any unexplained finding.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. terraform init
    Downloads the provider. Run it once per folder.
  3. terraform plan
    Shows what will be created. Changes nothing. Read it.
  4. terraform apply
    Shows the plan again and waits for you to type yes.
  5. terraform output verify
    Prints the commands that prove it worked.
Download all six files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 73 checks passed, and none flagged.

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/archive 2.8.1, hashicorp/aws 6.66.0.

outcomes/evidence-capture/aws/t2
.checkov.yml
Open .checkov.yml as plain text
guard.yml
Open guard.yml as plain text
main.tf
Open main.tf as plain text
outputs.tf
Open outputs.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text

How to check it worked

Run terraform output verify on a test instance you do not mind isolating. Tag it incident=drill-1, then check for the snapshots, the quarantine group, and, in Standard, the manifest and memory image in the evidence bucket. Termination must be refused.

What catches people

The instance uploads its own memory image, using its own role, which will not mention the evidence bucket. The bucket policy lets any instance in the account write under memory/ and read the capture tool, and nothing else. The ec2:SourceInstanceARN condition limits that to requests signed with an instance's own credentials.

Every parameter of the capture document has an allowed pattern. Each one ends up inside a shell script that runs as root. Without the patterns, whoever can run the document can run anything on any managed instance.

Isolation waits for memory capture, because the capture uploads over the network. If Systems Manager cannot reach the instance, isolation happens straight away and the status message says why.

How to undo it

terraform destroy removes the quick start. The snapshots it took stay until you delete them, which is intended. The Standard evidence bucket has prevent_destroy and Object Lock, so evidence outlives the tooling that collected it. To release an instance, move it back to its own security groups, turn termination protection off, and remove the tag.

What it costs

Nothing while idle. Each incident costs snapshot storage for its volumes and the memory image in S3.

Good to know

  • Check what you downloaded with evidence-verify, a small Go program on the Verifying evidence page. It recomputes every file's SHA-256 against the hash recorded beside it at capture, and writes a report for the case notes.
  • The order follows the order of volatility, with one twist. Memory disappears first, so it would normally be captured first. Writing a memory image changes the disk, though. An EBS snapshot is fixed at the instant you request it, so asking for the snapshot first costs the memory capture a few seconds and keeps the disk evidence clean.
  • Changing security groups does not always end connections that are already open. Tracked connections can keep flowing until they time out. If they must stop immediately, add a network ACL deny for the subnet as well, knowing it affects every instance in that subnet.
  • Moving evidence to a separate forensics account keeps it out of reach of whoever compromised this one. Snapshots encrypted with the default EBS key cannot be shared, so volumes need a customer-managed key for that.
  • Run a drill on a test instance every quarter. The first time this runs should never be the real incident.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.