iac.htora.dev · security templates

Home/For responders/Incident response on AWS

Incident response on AWS

AWS keeps 90 days of changes to your account without being asked, and nothing about reads of your data unless you set that up. What you can reconstruct depends on the trail you built before the incident.

1An alert or a reporta GuardDuty finding, an odd bill, a tip2Scope from CloudTrailwhich key or role, which accounts, since when3Contain by what was compromiseda key, a role's sessions, an Identity Center user, an instance4Preserve before you rebuildsnapshots and memory, hashed and locked
The first hour on AWS. Preservation runs alongside containment, never after a rebuild.

What AWS records, and what it leaves out

RecordOn by defaultKept
Event historyYes, with no trail: management events, per Region90 days
A trail or event data storeManagement events once you create one. Data events, such as S3 object reads, only for the resource types you add, at extra costAs long as you keep the files, or 366 days by default in CloudTrail Lake, extendable to 3,653
Network activity eventsNo. Calls made through your VPC endpoints, including those the endpoint policy refused, once you add them per serviceAs for the trail

Management events answer "who changed this": a new access key, a new role, a changed trust policy. Data events answer "who read this". With S3 data events off, copying every object out of a bucket leaves no CloudTrail record. Switch them on for the buckets that contain sensitive data before you need them. The audit log template keeps the trail where nobody in the account can delete it.

Reading an identity out of CloudTrail

  • The key tells you the kind of credential. userIdentity.accessKeyId starting AKIA is a long-term IAM user key. ASIA is a temporary key from STS, such as a role session.
  • A temporary key leads back to where it was issued. The AssumeRole event records the new key in responseElements.credentials.accessKeyId. Search for that key to find the session's start, and who started it.
  • A role session names its role. userIdentity.sessionContext.sessionIssuer is the role. sourceIdentity, when your roles require it, names the person, stays the same through every role chained after it, and cannot be changed by the session.
  • Identity Center users sign in through their own events. CredentialChallenge, CredentialVerification, and UserAuthentication, tied together by one AuthWorkflowID. Their work in an account then appears as an AssumedRole on a role named AWSReservedSSO_<permission set>_<id>, with the user's name as the session name.
aws cloudtrail lookup-events --max-results 50 \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=<key>
aws cloudtrail lookup-events --start-time <date> \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<user or session name>

Event history is per Region. Look in every Region the account uses, and in the ones it should not use.

Containment, by what was compromised

The containment refresher puts the first move for each kind of compromise on one printable page. The commands below follow the AWS CLI reference; replace each value in angle brackets.

CompromisedFirst moveWatch out
An IAM user's access keyDeactivate the key. Deactivating can be undone; delete it once the investigation no longer needs it
aws iam update-access-key --user-name <user> \
  --access-key-id <key> --status Inactive
A second key, a console password, and anything the key created, such as a new user or role
A role's sessionsRevoke active sessions on the role, in the console or with this inline policy. It denies every credential issued before the time you write in it
aws iam put-role-policy --role-name <role> \
  --policy-name RevokeOlderSessions \
  --policy-document file://revoke.json
{"Version": "2012-10-17", "Statement": {"Effect": "Deny",
 "Action": "*", "Resource": "*", "Condition": {"DateLessThan":
 {"aws:TokenIssueTime": "<now, as 2026-09-01T12:00:00Z>"}}}}
It does not work on service-linked roles. Roles from Identity Center permission sets are revoked through Identity Center
An Identity Center userFind the user's ID and add it to a deny policy in a service control policy prepared beforehand. Then, in the Identity Center console, delete their active sessions and disable the user
aws identitystore get-user-id --identity-store-id <d-id> \
  --alternate-identifier '{"UniqueAttribute": {"AttributePath": "userName", "AttributeValue": "<user>"}}'
aws organizations update-policy --policy-id <p-id> \
  --content file://deny.json
Role sessions already issued last up to the permission set's session length, at most 12 hours. Keep the deny in place for 12 hours. Delete the sessions before disabling the user; after that, the console can no longer stop them
An EC2 instanceTermination protection, snapshots, memory with your own tool, then a quarantine security group
aws ec2 modify-instance-attribute --instance-id <id> \
  --disable-api-termination
aws ec2 create-snapshots --copy-tags-from-source volume \
  --instance-specification InstanceId=<id>
aws ec2 modify-instance-attribute --instance-id <id> \
  --groups <quarantine-sg>
Changing a security group does not interrupt connections already open. A network ACL does, for the whole subnet
An instance role's credentials, used elsewhereRevoke the role's older sessions, with the same inline policy on the instance role
aws iam put-role-policy --role-name <instance role> \
  --policy-name RevokeOlderSessions \
  --policy-document file://revoke.json
GuardDuty reports this as InstanceCredentialExfiltration, OutsideAWS or InsideAWS

A deny in a service control policy stops every principal in the account, including its root user, and never grants anything. It has no effect on the management account or on service-linked roles.

Saved queries

16 saved queries from four templates open in this cloud's console, named as the console lists them. The Standard and Hardened levels install them; the Quick start installs none. The CloudWatch ones are in the Logs Insights query language, and the Athena ones in SQL.

Saved queryWhat it answersTemplate that installs itWhere to open it
ai-model-logging/tokens-by-callerCallers ranked by input and output tokens, from the invocation records. This is the query AWS publishes.Know who is using your AI modelsCloudWatch console, Logs Insights, Queries, folder ai-model-logging
ai-model-logging/calls-by-model-and-callerCalls and tokens by model and principal.Know who is using your AI modelsCloudWatch console, Logs Insights, Queries, folder ai-model-logging
ai-model-logging/one-callerWhat one caller asked and what came back. Replace the name in the filter first.Know who is using your AI modelsCloudWatch console, Logs Insights, Queries, folder ai-model-logging
ai-model-logging/refused-calls-by-callerRefused Bedrock calls by principal and error code, as when someone tests stolen keys. Reads the trail, so it covers every Region.Know who is using your AI modelsCloudWatch console, Logs Insights, Queries, folder ai-model-logging
ai-model-logging/bearer-token-callsBedrock calls made with a Bedrock API key. Reads the trail, so it covers every Region.Know who is using your AI modelsCloudWatch console, Logs Insights, Queries, folder ai-model-logging
insider-data-theft/permission-changesWho gave whom new IAM access, newest first. Run it first when an alarm names a person.Notice data walking out the doorCloudWatch console, Logs Insights, Queries, folder insider-data-theft
insider-data-theft/one-principalEverything one principal did. Replace the name in the filter first.Notice data walking out the doorCloudWatch console, Logs Insights, Queries, folder insider-data-theft
insider-data-theft/changes-by-personThe changes that alarm, with who made each one, for reviewing a week at once.Notice data walking out the doorCloudWatch console, Logs Insights, Queries, folder insider-data-theft
insider-data-theft/decrypt-grantsKMS grants that let someone decrypt, made with no AWS service acting for the caller.Notice data walking out the doorCloudWatch console, Logs Insights, Queries, folder insider-data-theft
insider-data-theft/bulk-readersWho read the most S3 objects, hour by hour.
Only when sensitive_bucket_arns is set.
Notice data walking out the doorCloudWatch console, Logs Insights, Queries, folder insider-data-theft
okta/one-actorEverything one Okta user did, newest first. Replace the login first.Catch account takeover at the identity provider
Okta
CloudWatch console, Logs Insights, Queries, folder okta
okta/admin-and-policy-changesChanges to who can sign in, how, and with what rights.Catch account takeover at the identity provider
Okta
CloudWatch console, Logs Insights, Queries, folder okta
okta/sign-ins-by-sourceSign-ins by user, address, and result, to spot a spray or a new address.Catch account takeover at the identity provider
Okta
CloudWatch console, Logs Insights, Queries, folder okta
okta/factor-changesWhose MFA factors changed, and who changed them.Catch account takeover at the identity provider
Okta
CloudWatch console, Logs Insights, Queries, folder okta
create-flow-log-tableCreates the flows table over the flow log files. Run it once, before the other.Keep network records long enough to be usefulAthena query editor, Saved queries, workgroup vpc-flow-logs
one-host-one-dayFlows and bytes for one address on one day. Edit the date and address first.Keep network records long enough to be usefulAthena query editor, Saved queries, workgroup vpc-flow-logs

Forensics on an EC2 instance

  • Keep it. AWS's incident response guide lists termination protection, shutdown behaviour set to stop, and DeleteOnTermination turned off on every volume. Termination protection alone does not stop an Auto Scaling group, or a shutdown from inside when shutdown behaviour is terminate, and Spot instances cannot take it.
  • Disks. aws ec2 create-snapshots snapshots every volume of an instance at one point in time. The EBS direct APIs read a snapshot's blocks without creating a volume.
  • Memory. AWS has no managed memory capture. The guide says to capture it with a tool of your own before isolating or shutting the instance down. The evidence template runs AVML through Systems Manager when someone tags the instance.
  • Where to look at it. A separate forensics account, with a workstation that cannot reach the internet. Snapshots under the default EBS key cannot be shared with it, so encrypt volumes with a key of your own.

Detection you can switch on

ServiceWhat it adds
GuardDutyReads CloudTrail management events, VPC flow logs, and DNS queries to AWS's own resolvers from its own copies, with nothing to switch on. Protection plans add S3 data events, EKS audit logs, runtime monitoring on EKS, ECS, and EC2, malware scans of volumes and new S3 objects, RDS logins, and Lambda network activity
GuardDuty Extended Threat DetectionOn by default at no extra cost. Joins findings and events over 24 hours into one critical attack sequence, such as compromised credentials or a compromised EKS cluster
Amazon DetectiveUp to a year of CloudTrail, flow log, EKS audit, and GuardDuty history, arranged for investigating one identity or resource
Security HubThe service AWS now calls Security Hub CSPM checks configuration. The newer Security Hub, generally available since December 2025, correlates GuardDuty, Inspector, Security Hub CSPM, and Macie findings into exposures
AWS Security Incident ResponseA paid service that triages GuardDuty and Security Hub CSPM findings, runs containment for EC2, S3, and IAM that you authorise in advance, and brings in AWS's incident responders

Containers on AWS

EKS adds a cluster audit log that is off until you switch it on, nodes that Auto Scaling and Karpenter replace, and Fargate pods with no node at all. Containers on GKE and EKS covers each one, and the container evidence template captures a compromised pod.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.