Home/Templates/Keep network records long enough to be useful
Keep network records long enough to be useful
Traffic records stored cheaply for months, in a format you can query.
Terraform
Why bother
Working out how far an attacker got needs network records from before you knew anything was wrong. Default retention is usually a few days, which is shorter than the time most intrusions go unnoticed.
Ask for traffic from a host thirty days ago and get rows back.
How it works
Set it up
Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.
Terraform. VPC flow logs to S3 in Parquet format
What you need first
- An account where you can create S3 buckets and flow logs.
- The VPC ID, or for Standard, several.
- Athena, to query the records.
What it creates
- Flow logs for every connection in the VPC, accepted or rejected, written in Parquet and partitioned by hour
- A bucket that keeps them a year, and the policy that lets the log service write there
- The Athena table statement, using partition projection so each hour is found with no loading step
- Extra fields for investigations, encryption under its own key, an Athena workgroup, and the table and a starter query saved in it (Standard)
Saved queries this installs
Standard and Hardened install these two, ready for the first hour of a case. Quick start installs none.
| Saved query | What it answers | Where to open it |
|---|---|---|
create-flow-log-table | Creates the flows table over the flow log files. Run it once, before the other. | Athena query editor, Saved queries, workgroup vpc-flow-logs |
one-host-one-day | Flows and bytes for one address on one day. Edit the date and address first. | Athena query editor, Saved queries, workgroup vpc-flow-logs |
The code
Quick startReady
vpc_id = "vpc-xxxxxxxxxxxxxxxxx"main.tf, line 15
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: 24 checks passed, and five rules flagged, each with a written reason:
- S3 event notifications
- S3 cross-region replication
- S3 encrypted with KMS
- S3 server access logging
- S3 versioning on the SIEM failed-events bucket
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open main.tf as plain text
StandardReady
vpc_ids. VPCs to record.
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.# terraform.tfvars vpc_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 51 checks passed, and seven rules flagged, each with a written reason:
- S3 event notifications
- IAM permissions management without constraints
- IAM write access without constraints
- S3 cross-region replication
- S3 server access logging
- S3 versioning on the SIEM failed-events bucket
- IAM resource of *
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
vpc_ids. VPCs to record.
aws sts get-caller-identity
Shows the account and role you are signed in as. Check it before every apply.# terraform.tfvars vpc_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: 48 checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/aws 6.66.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
After an hour of traffic, create the table from terraform output athena_table, then run terraform output example_query in Athena. Rows come back. Standard uses saved queries in its own workgroup; terraform output verify prints the steps.
With hour partitions on, files land under AWSLogs/aws-account-id=<id>/, and a bucket policy written for AWSLogs/<id>/ refuses them. Both levels allow both paths.
The default fields show the network interface's address, which behind a NAT gateway or load balancer is not the real source. Standard adds pkt-srcaddr and pkt-dstaddr for the addresses behind it.
Flow logs show who talked to whom, when, and how much. What was said needs a packet capture.
Records arrive every few minutes, so a query about the last ten minutes can come back empty.
How to undo it
terraform destroy stops the flow logs and deletes the bucket and records. Standard's bucket has no force delete, so destroy stops there until you empty it.
What it costs
Flow logs delivered to S3 are charged per gigabyte, and Parquet adds a small conversion charge. Athena charges per terabyte scanned, which partitions keep low.
Terraform. Virtual network flow logs with traffic analytics
What you need first
- A virtual network, and Network Contributor on it.
- Network Watcher in the network's region, which Azure creates in NetworkWatcherRG with the first network there.
What it creates
- A flow log for one virtual network, kept 90 days in a storage account
- Traffic analytics summarising it every ten minutes in a Log Analytics workspace
- Several networks, zone-redundant storage moved to the cool tier after 30 days, and your own retention (Standard)
- Saved searches for flagged flows, busiest internet destinations, and everything one address talked to (Standard)
Saved queries this installs
Standard and Hardened install these three, ready for the first hour of a case. Quick start installs none.
| Saved query | What it answers | Where to open it |
|---|---|---|
malicious-flows | Flows to or from addresses Microsoft rates malicious. | Log Analytics workspace, Logs, Queries, category Investigation |
top-outbound-destinations | Busiest outbound destinations on the internet, last day. | Log Analytics workspace, Logs, Queries, category Investigation |
one-address | Everything one private address talked to. Replace the address first. | Log Analytics workspace, Logs, Queries, category Investigation |
The code
Quick startReady
vnet_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network/providers/Microsoft.Network/virtualNetworks/vnet-main"main.tf, line 20location = "eastus"main.tf, line 21
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: eight checks passed, and seven rules flagged, each with a written reason:
- Storage encrypted with your own key
- Storage private endpoint
- Storage soft delete
- Storage account shared key access off
- Storage replication
- Classic queue logging
- Storage public network access off
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open main.tf as plain text
StandardReady
subscription_id. Subscription for the storage account and workspace.vnet_ids. Virtual networks to log, by a short name of your choosing, all in var.location.
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.# terraform.tfvars subscription_id = "" vnet_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: nine checks passed, and six rules flagged, each with a written reason:
- Storage encrypted with your own key
- Storage private endpoint
- Storage account shared key access off
- Storage replication
- Classic queue logging
- Storage public network access off
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
subscription_id. Subscription for the storage account and workspace.vnet_ids. Virtual networks to log, by a short name of your choosing, all in var.location.
az login
Signs the command line tool in. Terraform uses the same sign-in.export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
Tells the provider which subscription. It stopped guessing in version 4.# terraform.tfvars subscription_id = "" vnet_ids = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: nine checks passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/azurerm 5.6.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
About ten minutes after traffic, flow records are in the storage account's insights-logs-flowlogflowevent container. About an hour after, NTANetAnalytics in the workspace has rows.
A flow log must be written by the Network Watcher in the network's own region. If NetworkWatcherRG was removed, create it first.
Traffic between a machine and a private endpoint is not always recorded. Check a known flow before relying on it.
The saved searches use NTANetAnalytics column names as Microsoft documents them. Check one against your workspace before depending on it.
How to undo it
terraform destroy removes the flow logs, storage, and workspace, with the records in them.
What it costs
Per GB of flow logs collected, per GB processed by traffic analytics, plus storage and Log Analytics.
Terraform. VPC flow logs to a log bucket
What you need first
- A project where you can create log buckets and sinks.
- The subnets to record, and rights to update them.
What it creates
- A log bucket that keeps flow logs a year, with Log Analytics so they can be queried with SQL
- A sink that routes flow logs into it
- The command that switches flow logs on for a subnet, and an example query
- Your own retention and sampling, one command per subnet, and a linked BigQuery dataset (Standard)
The code
Quick startReady
project_id = "your-project-id"main.tf, line 15region = "us-central1"main.tf, line 16
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: one check passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open main.tf as plain text
StandardReady
project_id. Project containing the subnets and the log bucket.
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.# terraform.tfvars project_id = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: one check passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
project_id. Project containing the subnets and the log bucket.
gcloud auth application-default login
The sign-in Terraform reads. A plain gcloud auth login is not enough.# terraform.tfvars project_id = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: one check passed, and none flagged.
Last run 2026-10-02 with OpenTofu v1.12.6 against hashicorp/google 8.4.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
Switch flow logs on for a subnet, wait for traffic, and run terraform output example_query in Log Analytics. Rows come back. Standard prints its checks with terraform output verify.
Flow logs are a setting on each subnet, and this template does not manage your subnets. Run the printed command for each one.
Sampling below 1.0 records a share of flows. A short connection can be missed entirely, so raise it for subnets that matter most in an investigation.
Flow logs also land in the _Default bucket for its 30 days, so the first month is stored twice. Exclude them from _Default in the Log Router if that cost matters.
How to undo it
terraform destroy removes the sink, the bucket, and its records. Subnets keep recording until you run the command with --no-enable-flow-logs.
What it costs
Flow logs are charged by volume generated, and sampling sets that volume. Storage beyond the default 30 days is billed per gigabyte per month.
Terraform. VCN flow logs to the Logging service
What you need first
- A subnet, and permission to manage logs in its compartment.
What it creates
- A log group and a flow log for one subnet, recording accepted and rejected connections, kept 90 days
- Several subnets, searchable for up to 180 days (Standard)
- A copy of every record in an Object Storage bucket whose retention rule stops deletes, 400 days by default (Standard)
- Saved searches for refused connections, refusals by source, and everything one address talked to (Standard)
Saved queries this installs
Standard and Hardened install these three, ready for the first hour of a case. Quick start installs none.
| Saved query | What it answers | Where to open it |
|---|---|---|
flow-logs-rejected | Connections the network refused, newest first. | Logging, Saved searches, in the compartment |
flow-logs-rejected-by-source | Refused connections counted by source address. | Logging, Saved searches, in the compartment |
flow-logs-one-address | Everything one address talked to. Replace the address first. | Logging, Saved searches, in the compartment |
The code
Quick startReady
compartment_ocid = "ocid1.compartment.oc1..change-me"main.tf, line 15subnet_ocid = "ocid1.subnet.oc1..change-me"main.tf, line 16
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform destroy
When you have finished. If it refuses, How to undo it below explains why.
Formatting passedValid passedLint passed
Security scan: one check passed, and none flagged.
Last run 2026-11-09 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open main.tf as plain text
StandardReady
region. Region of the subnets, such as us-ashburn-1.compartment_ocid. Compartment for the log group, archive bucket, connector, and saved searches.subnets. Subnets to log, by a short name of your choosing, all in var.region.
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.# terraform.tfvars region = "" compartment_ocid = "" subnets = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: two checks passed, and three rules flagged, each with a written reason:
Last run 2026-11-09 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
HardenedReady
region. Region of the subnets, such as us-ashburn-1.compartment_ocid. Compartment for the log group, archive bucket, connector, and saved searches.subnets. Subnets to log, by a short name of your choosing, all in var.region.
oci os ns get
Prints your Object Storage namespace, which proves ~/.oci/config works.# terraform.tfvars region = "" compartment_ocid = "" subnets = ""
Create this file beside the others and fill in your values. Terraform reads it on its own.terraform init
Downloads the provider. Run it once per folder.terraform plan
Shows what will be created. Changes nothing. Read it.terraform apply
Shows the plan again and waits for you to type yes.terraform output verify
Prints the commands that prove it worked.
Formatting passedValid passedLint passed
Security scan: two checks passed, and none flagged.
Last run 2026-11-09 with OpenTofu v1.12.6 against oracle/oci 9.8.0.
Open .checkov.yml as plain text
Open guard.yml as plain text
Open main.tf as plain text
Open outputs.tf as plain text
Open variables.tf as plain text
Open versions.tf as plain text
How to check it worked
A few minutes after traffic, search the log group in Logging Search for data.action = 'REJECT'. Refused connections are listed.
Logging keeps records for 180 days at most. Anything longer needs the Standard archive.
Flow logs record connections, never their contents.
The archive's retention rule is unlocked so you can test it. Lock it in the console once the period is right; a locked rule cannot be shortened.
Deleting Standard's connector policy, or moving the connector out of its compartment, stops the archive copy, and nothing reports it. The daily drift plan in Hardened is what notices.
How to undo it
terraform destroy removes the logs and searches. Standard's archive bucket cannot be deleted while its retention rule protects objects.
What it costs
Logging bills per GB stored beyond the free allowance. Object Storage bills per GB kept.
Good to know
- Flow logs answer the first questions of a network investigation: which hosts a compromised machine talked to, on which ports, and how much data left.
- Keep them longer than your other logs if you can. Intrusions are often found months after they begin, and flow records are cheap next to what they explain.