iac.htora.dev · security templates

Home/Templates/Audit logs nobody can delete

Audit logs nobody can delete

Send cloud audit logs to storage that refuses deletes for a set number of days.

Terraform

Why bother

An audit log only helps if it is still there after an incident. In a normal setup anyone with admin rights can delete it, and that is often the first thing someone does to cover their tracks. This sends the logs to storage with a retention lock, so deletes fail for everyone, including the account owner.

How you know it worked

Try to delete a stored log file. The delete should fail. The exact commands differ by platform, and each tab below spells them out.

Set it up

Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.

Terraform. CloudTrail writing to S3 with Object Lock turned on

What you need first

  • An account where you can create S3 buckets, KMS keys, and CloudTrail trails.
  • For the organization option, run it in the management account with CloudTrail trusted access already on.

What it creates

  • A bucket with Object Lock, versioning, and public access blocked
  • A trail covering every region, with log file validation on
  • A KMS key with yearly rotation (Standard only)
  • A CloudWatch log group so metric alarms have something to read (Standard, optional)

The code

Quick startDraft1 file, 234 lines
One file. Bucket, one-day lock, policy, trail. Encryption is AWS-managed. Built to try in a scratch account and remove the next day.
outcomes/audit-log-immutable/aws/t0
main.tf
# audit-log-immutable / aws / t0 "Paste"
#
# CloudTrail writing to an S3 bucket with Object Lock in GOVERNANCE mode.
# A locked log file cannot be deleted until its retention period ends.
#
# Run:
#   terraform init && terraform apply
#
# Verify, once the first logs arrive (about fifteen minutes):
#   terraform output verify
#   Run the two commands it prints. The second must fail with AccessDenied.
#
# Why two commands: the bucket is versioned, so a plain delete-object only adds
# a "delete marker" and reports success while the log file is still there. The
# lock protects each stored version, so the real test deletes a version by ID.
#
# Retention here is one day, so you can try this in a scratch account and tear
# it down tomorrow. The Standard version defaults to 400 days.

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {}

data "aws_caller_identity" "current" {}
data "aws_partition" "current" {}

resource "random_id" "suffix" {
  byte_length = 4
}

locals {
  bucket_name    = "audit-log-immutable-${data.aws_caller_identity.current.account_id}-${random_id.suffix.hex}"
  trail_name     = "audit-log-immutable"
  retention_days = 1
}

# ---------------------------------------------------------------------------
# Bucket. object_lock_enabled can only be set at creation time.
# ---------------------------------------------------------------------------
resource "aws_s3_bucket" "audit" {
  bucket              = local.bucket_name
  object_lock_enabled = true

  # True here so terraform destroy can empty the bucket once the one-day lock
  # has passed. The Standard version sets this false.
  force_destroy = true

  tags = {
    owner               = "security-engineering"
    environment         = "prod"
    data_classification = "restricted"
    managed_by          = "terraform"
  }
}

# Object Lock requires versioning. Terraform will not enable it implicitly.
resource "aws_s3_bucket_versioning" "audit" {
  bucket = aws_s3_bucket.audit.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "audit" {
  bucket     = aws_s3_bucket.audit.id
  depends_on = [aws_s3_bucket_versioning.audit]

  rule {
    default_retention {
      mode = "GOVERNANCE"
      days = local.retention_days
    }
  }
}

resource "aws_s3_bucket_public_access_block" "audit" {
  bucket                  = aws_s3_bucket.audit.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "audit" {
  bucket = aws_s3_bucket.audit.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "audit" {
  bucket     = aws_s3_bucket.audit.id
  depends_on = [aws_s3_bucket_versioning.audit]

  rule {
    id     = "abort-incomplete-uploads"
    status = "Enabled"
    filter {}
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

# ---------------------------------------------------------------------------
# Bucket policy. Both statements are required or CreateTrail fails validation.
# aws:SourceArn scopes delivery to this trail only.
# ---------------------------------------------------------------------------
data "aws_iam_policy_document" "audit" {
  statement {
    sid     = "AWSCloudTrailAclCheck"
    effect  = "Allow"
    actions = ["s3:GetBucketAcl"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    resources = [aws_s3_bucket.audit.arn]
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid     = "AWSCloudTrailWrite"
    effect  = "Allow"
    actions = ["s3:PutObject"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    resources = ["${aws_s3_bucket.audit.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"]
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid     = "DenyUnencryptedTransport"
    effect  = "Deny"
    actions = ["s3:*"]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    resources = [
      aws_s3_bucket.audit.arn,
      "${aws_s3_bucket.audit.arn}/*",
    ]
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

resource "aws_s3_bucket_policy" "audit" {
  bucket = aws_s3_bucket.audit.id
  policy = data.aws_iam_policy_document.audit.json

  # Applying a policy while the public access block is still being created can
  # fail intermittently. Ordering them removes the race.
  depends_on = [aws_s3_bucket_public_access_block.audit]
}

# ---------------------------------------------------------------------------
# Trail. The ARN is built from parts to avoid a cycle:
# the bucket policy needs the trail ARN, the trail needs the bucket policy.
# ---------------------------------------------------------------------------
locals {
  trail_arn = "arn:${data.aws_partition.current.partition}:cloudtrail:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:trail/${local.trail_name}"
}

data "aws_region" "current" {}

resource "aws_cloudtrail" "audit" {
  name                          = local.trail_name
  s3_bucket_name                = aws_s3_bucket.audit.id
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true

  depends_on = [aws_s3_bucket_policy.audit]

  tags = {
    owner               = "security-engineering"
    environment         = "prod"
    data_classification = "restricted"
    managed_by          = "terraform"
  }
}

output "bucket" {
  description = "Bucket holding the trail. Objects are WORM protected for the retention period."
  value       = aws_s3_bucket.audit.id
}

output "trail_arn" {
  description = "Trail ARN."
  value       = aws_cloudtrail.audit.arn
}

output "verify" {
  description = "Two commands. The second must fail with AccessDenied."
  value       = <<-EOT
    aws s3api list-object-versions --bucket ${aws_s3_bucket.audit.id} --prefix AWSLogs/ --max-items 1 --query 'Versions[0].[Key,VersionId]' --output text
    aws s3api delete-object --bucket ${aws_s3_bucket.audit.id} --key <Key from above> --version-id <VersionId from above>
  EOT
}
StandardDraft5 files, 479 lines
Adds your own KMS key, the organization trail switch, CloudWatch delivery, lifecycle to cold storage, and a deny on anyone changing the lock.
outcomes/audit-log-immutable/aws/t1
kms.tf
# Customer-managed key. Two reasons over SSE-S3: rotation you control, and a
# revocation lever that does not require deleting data.

resource "aws_kms_key" "audit" {
  description             = "${var.name} trail encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true
  policy                  = data.aws_iam_policy_document.key.json
}

resource "aws_kms_alias" "audit" {
  name          = "alias/${var.name}"
  target_key_id = aws_kms_key.audit.key_id
}

data "aws_iam_policy_document" "key" {
  statement {
    sid    = "AccountRootManagesKey"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:${data.aws_partition.current.partition}:iam::${local.account_id}:root"]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }

  statement {
    sid    = "CloudTrailEncrypt"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    actions   = ["kms:GenerateDataKey*", "kms:DescribeKey"]
    resources = ["*"]
    condition {
      test     = "StringLike"
      variable = "kms:EncryptionContext:aws:cloudtrail:arn"
      values   = ["arn:${data.aws_partition.current.partition}:cloudtrail:*:${local.account_id}:trail/*"]
    }
  }

  # CloudWatch Logs encrypts the log group with this key, and it can only do that
  # if the key policy names it. Without this statement the log group fails to
  # create with a message about the key not existing.
  statement {
    sid    = "CloudWatchLogsEncrypt"
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["logs.${data.aws_region.current.name}.amazonaws.com"]
    }
    actions = [
      "kms:Encrypt*",
      "kms:Decrypt*",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:Describe*",
    ]
    resources = ["*"]
    condition {
      test     = "ArnLike"
      variable = "kms:EncryptionContext:aws:logs:arn"
      values   = ["arn:${data.aws_partition.current.partition}:logs:${data.aws_region.current.name}:${local.account_id}:log-group:/aws/cloudtrail/${var.name}"]
    }
  }

  statement {
    sid    = "ReadersDecrypt"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:${data.aws_partition.current.partition}:iam::${local.account_id}:root"]
    }
    actions   = ["kms:Decrypt", "kms:ReEncryptFrom"]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "kms:CallerAccount"
      values   = [local.account_id]
    }
  }
}
main.tf
# audit-log-immutable / aws / t1 "Standard"
#
# CloudTrail to an Object Lock bucket, encrypted with a customer-managed key,
# optionally as an organization trail, optionally mirrored to CloudWatch Logs.
#
# Verify, once logs arrive:
#   terraform output verify
#   The second command it prints must fail with AccessDenied. A delete without a
#   version ID only adds a delete marker and succeeds, which proves nothing.

data "aws_caller_identity" "current" {}
data "aws_partition" "current" {}
data "aws_region" "current" {}

locals {
  account_id = data.aws_caller_identity.current.account_id
  partition  = data.aws_partition.current.partition

  tags = merge({
    owner               = var.owner
    environment         = var.environment
    data_classification = "restricted"
    managed_by          = "terraform"
  }, var.extra_tags)

  # Built from parts to avoid a cycle. The bucket policy needs the trail ARN and
  # the trail needs the bucket policy, so a direct reference is a cycle.
  trail_arn = "arn:${local.partition}:cloudtrail:${data.aws_region.current.name}:${local.account_id}:trail/${var.name}"

  # Organization trails write under AWSLogs/<org-id>/, single-account trails
  # under AWSLogs/<account-id>/.
  log_prefixes = var.organization_trail ? [
    "${aws_s3_bucket.audit.arn}/AWSLogs/${local.account_id}/*",
    "${aws_s3_bucket.audit.arn}/AWSLogs/${data.aws_organizations_organization.current[0].id}/*",
    ] : [
    "${aws_s3_bucket.audit.arn}/AWSLogs/${local.account_id}/*",
  ]
}

data "aws_organizations_organization" "current" {
  count = var.organization_trail ? 1 : 0
}

# ---------------------------------------------------------------------------
# Storage
# ---------------------------------------------------------------------------
resource "aws_s3_bucket" "audit" {
  bucket              = "${var.name}-${local.account_id}"
  object_lock_enabled = true
  force_destroy       = false
}

resource "aws_s3_bucket_versioning" "audit" {
  bucket = aws_s3_bucket.audit.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "audit" {
  bucket     = aws_s3_bucket.audit.id
  depends_on = [aws_s3_bucket_versioning.audit]

  rule {
    default_retention {
      mode = var.lock_mode
      days = var.retention_days
    }
  }
}

resource "aws_s3_bucket_public_access_block" "audit" {
  bucket                  = aws_s3_bucket.audit.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "audit" {
  bucket = aws_s3_bucket.audit.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.audit.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "audit" {
  bucket     = aws_s3_bucket.audit.id
  depends_on = [aws_s3_bucket_versioning.audit]

  rule {
    id     = "cold-storage-after-90-days"
    status = "Enabled"
    filter {}
    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }
  }

  rule {
    id     = "abort-incomplete-uploads"
    status = "Enabled"
    filter {}
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

data "aws_iam_policy_document" "bucket" {
  statement {
    sid     = "AWSCloudTrailAclCheck"
    effect  = "Allow"
    actions = ["s3:GetBucketAcl"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    resources = [aws_s3_bucket.audit.arn]
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid       = "AWSCloudTrailWrite"
    effect    = "Allow"
    actions   = ["s3:PutObject"]
    resources = local.log_prefixes
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }

  statement {
    sid     = "DenyUnencryptedTransport"
    effect  = "Deny"
    actions = ["s3:*"]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    resources = [aws_s3_bucket.audit.arn, "${aws_s3_bucket.audit.arn}/*"]
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }

  # Skipped until terraform_role_name is set. Applying it on the first run
  # would deny the very principal creating the bucket.
  dynamic "statement" {
    for_each = var.terraform_role_name == "" ? [] : [1]
    content {
      sid    = "DenyLifecycleAndLockTampering"
      effect = "Deny"
      actions = [
        "s3:PutBucketObjectLockConfiguration",
        "s3:PutLifecycleConfiguration",
        "s3:PutBucketVersioning",
      ]
      principals {
        type        = "AWS"
        identifiers = ["*"]
      }
      resources = [aws_s3_bucket.audit.arn]
      condition {
        test     = "ArnNotLike"
        variable = "aws:PrincipalArn"
        values   = ["arn:${local.partition}:iam::${local.account_id}:role/${var.terraform_role_name}"]
      }
    }
  }
}

resource "aws_s3_bucket_policy" "audit" {
  bucket     = aws_s3_bucket.audit.id
  policy     = data.aws_iam_policy_document.bucket.json
  depends_on = [aws_s3_bucket_public_access_block.audit]
}

# ---------------------------------------------------------------------------
# CloudWatch Logs delivery. Optional, and the reason alarms can exist at all.
# ---------------------------------------------------------------------------
resource "aws_cloudwatch_log_group" "audit" {
  count             = var.cloudwatch_logs ? 1 : 0
  name              = "/aws/cloudtrail/${var.name}"
  retention_in_days = var.cloudwatch_retention_days
  kms_key_id        = aws_kms_key.audit.arn
}

data "aws_iam_policy_document" "trail_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }
}

resource "aws_iam_role" "trail" {
  count              = var.cloudwatch_logs ? 1 : 0
  name               = "${var.name}-cloudwatch-delivery"
  assume_role_policy = data.aws_iam_policy_document.trail_assume.json
}

data "aws_iam_policy_document" "trail_delivery" {
  count = var.cloudwatch_logs ? 1 : 0
  statement {
    effect    = "Allow"
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.audit[0].arn}:*"]
  }
}

resource "aws_iam_role_policy" "trail_delivery" {
  count  = var.cloudwatch_logs ? 1 : 0
  name   = "delivery"
  role   = aws_iam_role.trail[0].id
  policy = data.aws_iam_policy_document.trail_delivery[0].json
}

# ---------------------------------------------------------------------------
# Trail
# ---------------------------------------------------------------------------
resource "aws_cloudtrail" "audit" {
  name                          = var.name
  s3_bucket_name                = aws_s3_bucket.audit.id
  kms_key_id                    = aws_kms_key.audit.arn
  include_global_service_events = true
  is_multi_region_trail         = true
  is_organization_trail         = var.organization_trail
  enable_log_file_validation    = true

  cloud_watch_logs_group_arn = var.cloudwatch_logs ? "${aws_cloudwatch_log_group.audit[0].arn}:*" : null
  cloud_watch_logs_role_arn  = var.cloudwatch_logs ? aws_iam_role.trail[0].arn : null

  depends_on = [aws_s3_bucket_policy.audit]
}
outputs.tf
output "bucket" {
  description = "Bucket holding the trail."
  value       = aws_s3_bucket.audit.id
}

output "trail_arn" {
  description = "Trail ARN."
  value       = aws_cloudtrail.audit.arn
}

output "kms_key_arn" {
  description = "Key encrypting the trail. Disabling it is the revocation lever."
  value       = aws_kms_key.audit.arn
}

output "lock_mode" {
  description = "GOVERNANCE or COMPLIANCE, as applied."
  value       = var.lock_mode
}

output "verify" {
  description = "Two commands. The second must fail with AccessDenied."
  value       = <<-EOT
    aws s3api list-object-versions --bucket ${aws_s3_bucket.audit.id} --prefix AWSLogs/ --max-items 1 --query 'Versions[0].[Key,VersionId]' --output text
    aws s3api delete-object --bucket ${aws_s3_bucket.audit.id} --key <Key from above> --version-id <VersionId from above>
  EOT
}
variables.tf
variable "name" {
  description = "Base name for the trail and bucket."
  type        = string
  default     = "audit-log-immutable"

  validation {
    condition     = can(regex("^[a-z0-9-]{3,40}$", var.name))
    error_message = "name must be 3-40 characters of lowercase letters, digits, or hyphens."
  }
}

variable "retention_days" {
  description = "Object Lock default retention. Set this to at least your longest expected dwell time."
  type        = number
  default     = 400

  validation {
    condition     = var.retention_days >= 90
    error_message = "retention_days must be 90 or more. Shorter than that, the log expires before most investigations start."
  }
}

variable "lock_mode" {
  description = "GOVERNANCE allows bypass by a principal holding s3:BypassGovernanceRetention. COMPLIANCE allows no bypass, including by the root user, for the full retention period."
  type        = string
  default     = "GOVERNANCE"

  validation {
    condition     = contains(["GOVERNANCE", "COMPLIANCE"], var.lock_mode)
    error_message = "lock_mode must be GOVERNANCE or COMPLIANCE."
  }
}

variable "organization_trail" {
  description = "Create an organization trail covering every member account. Requires the management or delegated administrator account with CloudTrail trusted access enabled."
  type        = bool
  default     = false
}

variable "cloudwatch_logs" {
  description = "Also deliver events to CloudWatch Logs so metric filters and alarms can run against them."
  type        = bool
  default     = true
}

variable "cloudwatch_retention_days" {
  description = "CloudWatch Logs retention. The S3 copy is the system of record, so this is the alerting window."
  type        = number
  default     = 90
}

variable "owner" {
  description = "Tag contract: accountable team."
  type        = string
  default     = "security-engineering"
}

variable "environment" {
  description = "Tag contract: environment."
  type        = string
  default     = "prod"
}

variable "extra_tags" {
  description = "Additional tags merged over the contract."
  type        = map(string)
  default     = {}
}

variable "terraform_role_name" {
  description = "IAM role name that Terraform assumes. When set, the bucket policy denies lock, lifecycle, and versioning changes to every other principal. Leave empty to skip that statement, which you should do on the first apply and then set."
  type        = string
  default     = ""
}
versions.tf
terraform {
  required_version = ">= 1.6"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }

  # Remote state. Uncomment and point at your own backend.
  # Native S3 locking (use_lockfile) needs Terraform 1.10 or later and removes
  # the DynamoDB table requirement.
  #
  # backend "s3" {
  #   bucket       = "tfstate-security-prod"
  #   key          = "audit-log-immutable/aws/terraform.tfstate"
  #   region       = "us-east-1"
  #   encrypt      = true
  #   kms_key_id   = "alias/tfstate"
  #   use_lockfile = true
  # }
}

provider "aws" {
  default_tags {
    tags = local.tags
  }
}
HardenedPlanned
Not written yet.

How to check it worked

Wait about fifteen minutes for the first logs, then run terraform output verify. It prints two commands. The first finds a stored log file and its version ID. The second tries to delete that version, and must fail with AccessDenied.

What catches people

Object Lock can only be switched on at the moment a bucket is created. You cannot add it to the log bucket you already have. Moving to this means a new bucket and a copy, which is worth knowing before you plan the work.

A plain delete-object on this bucket reports success. The bucket is versioned, so that call only adds a delete marker, and the locked file is still there underneath it. The lock protects each stored version, so the only honest test deletes a version by its ID.

How to undo it

The quick start locks for one day, so after that terraform destroy empties and removes the bucket. The Standard version locks for 400 days. In GOVERNANCE mode a principal holding s3:BypassGovernanceRetention can delete sooner. In COMPLIANCE mode nobody can, and you wait it out.

What it costs

Storage for the logs, plus about a dollar a month for the KMS key. The first management trail in an account carries no CloudTrail charge.

Registry 0.6.0. Built 2026-09-22.

Made by Habibullah Tora. Code under the MIT licence, writing under CC BY 4.0.