iac.htora.dev · security templates

Home/Templates/Keep network records long enough to be useful

Keep network records long enough to be useful

Traffic records stored cheaply for months, in a format you can query.

Terraform

Why bother

Working out how far an attacker got needs network records from before you knew anything was wrong. Default retention is usually a few days, which is shorter than the time most intrusions go unnoticed.

How you know it worked

Ask for traffic from a host thirty days ago and get rows back.

How it works

1Traffic crosses the networkbetween hosts and out to the internet2A flow record is writtenwho talked to whom, when, and how much3Stored cheaply for monthsin a format you can query4A scoping question answeredmonths after the fact
Step by step, from the thing that happens to the result you want.

Set it up

Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.

Terraform. VPC flow logs to S3 in Parquet format

What you need first

  • An account where you can create S3 buckets and flow logs.
  • The VPC ID, or for Standard, several.
  • Athena, to query the records.

What it creates

  • Flow logs for every connection in the VPC, accepted or rejected, written in Parquet and partitioned by hour
  • A bucket that keeps them a year, and the policy that lets the log service write there
  • The Athena table statement, using partition projection so each hour is found with no loading step
  • Extra fields for investigations, encryption under its own key, an Athena workgroup, and the table and a starter query saved in it (Standard)

Saved queries this installs

Standard and Hardened install these two, ready for the first hour of a case. Quick start installs none.

Saved queryWhat it answersWhere to open it
create-flow-log-tableCreates the flows table over the flow log files. Run it once, before the other.Athena query editor, Saved queries, workgroup vpc-flow-logs
one-host-one-dayFlows and bytes for one address on one day. Edit the date and address first.Athena query editor, Saved queries, workgroup vpc-flow-logs

The code

Quick startReadyone file, 179 lines
One file. One VPC in Parquet by the hour, kept a year, with the Athena table and an example query printed.
Change these first
  • vpc_id = "vpc-xxxxxxxxxxxxxxxxx" main.tf, line 15
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. terraform init
    Downloads the provider. Run it once per folder.
  3. terraform plan
    Shows what will be created. Changes nothing. Read it.
  4. terraform apply
    Shows the plan again and waits for you to type yes.
  5. terraform destroy
    When you have finished. If it refuses, How to undo it below explains why.
Download main.tf
Checks

Formatting passedValid passedLint passed

Security scan: 24 checks passed, and five rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/network-flow-logs/aws/t0
main.tf
Open main.tf as plain text
StandardReadyfour files, 328 lines
Adds several VPCs, investigation fields, encryption under its own key, cheaper storage after 30 days, and an Athena workgroup with the table and a starter query saved.
Change these first
  • vpc_ids. VPCs to record.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. # terraform.tfvars
    vpc_ids = ""
    Create this file beside the others and fill in your values. Terraform reads it on its own.
  3. terraform init
    Downloads the provider. Run it once per folder.
  4. terraform plan
    Shows what will be created. Changes nothing. Read it.
  5. terraform apply
    Shows the plan again and waits for you to type yes.
  6. terraform output verify
    Prints the commands that prove it worked.
Download all four files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 51 checks passed, and seven rules flagged, each with a written reason:

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/network-flow-logs/aws/t1
main.tf
Open main.tf as plain text
outputs.tf
Open outputs.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text
HardenedReadysix files, 201 lines
Standard as a module. Plan rules refuse the bucket or key deleted, force_destroy, flow logs not recording all traffic, and investigation fields dropped. They also refuse public access allowed, encryption other than KMS, key rotation off, and an Athena workgroup that stops enforcing encryption. A daily plan reports drift, and the scan fails on any unexplained finding.
Change these first
  • vpc_ids. VPCs to record.
Run it
  1. aws sts get-caller-identity
    Shows the account and role you are signed in as. Check it before every apply.
  2. # terraform.tfvars
    vpc_ids = ""
    Create this file beside the others and fill in your values. Terraform reads it on its own.
  3. terraform init
    Downloads the provider. Run it once per folder.
  4. terraform plan
    Shows what will be created. Changes nothing. Read it.
  5. terraform apply
    Shows the plan again and waits for you to type yes.
  6. terraform output verify
    Prints the commands that prove it worked.
Download all six files as a zip
Checks

Formatting passedValid passedLint passed

Security scan: 48 checks passed, and none flagged.

Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.

outcomes/network-flow-logs/aws/t2
.checkov.yml
Open .checkov.yml as plain text
guard.yml
Open guard.yml as plain text
main.tf
Open main.tf as plain text
outputs.tf
Open outputs.tf as plain text
variables.tf
Open variables.tf as plain text
versions.tf
Open versions.tf as plain text

How to check it worked

After an hour of traffic, create the table from terraform output athena_table, then run terraform output example_query in Athena. Rows come back. Standard uses saved queries in its own workgroup; terraform output verify prints the steps.

What catches people

With hour partitions on, files land under AWSLogs/aws-account-id=<id>/, and a bucket policy written for AWSLogs/<id>/ refuses them. Both levels allow both paths.

The default fields show the network interface's address, which behind a NAT gateway or load balancer is not the real source. Standard adds pkt-srcaddr and pkt-dstaddr for the addresses behind it.

Flow logs show who talked to whom, when, and how much. What was said needs a packet capture.

Records arrive every few minutes, so a query about the last ten minutes can come back empty.

How to undo it

terraform destroy stops the flow logs and deletes the bucket and records. Standard's bucket has no force delete, so destroy stops there until you empty it.

What it costs

Flow logs delivered to S3 are charged per gigabyte, and Parquet adds a small conversion charge. Athena charges per terabyte scanned, which partitions keep low.

Good to know

  • Flow logs answer the first questions of a network investigation: which hosts a compromised machine talked to, on which ports, and how much data left.
  • Keep them longer than your other logs if you can. Intrusions are often found months after they begin, and flow records are cheap next to what they explain.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.