Home/For responders/Leading a cloud incident
Leading a cloud incident
One-page refresher: Leading a cloud incident: the first hour One-page refresher: Containment by what was compromised
The phases are the ones every incident has. What changes in the cloud is how fast evidence disappears, how many ways an attacker can come back, and how much of the response is API calls.
What is different in the cloud
Compute disappears on its own. Autoscaling replaces unhealthy instances, spot capacity is reclaimed, and pods are rescheduled. Evidence has to be captured on purpose, early, or it is gone.
Identity is the way in. Most cloud intrusions are valid credentials making legitimate API calls. There is often no malware to find, only a key or a token used by the wrong person.
Everything is an API call. That makes the control-plane audit log the main source of evidence, and it makes containment something you can script and rehearse. Evidence that survives termination turns preservation into one tag.
Scope before you call it contained
An attacker with working credentials rarely stops at using them. They create ways back in, so that revoking the first credential changes nothing. Look for these before declaring the incident contained:
| Look for | AWS CloudTrail | Google Cloud audit logs | Azure |
|---|---|---|---|
| New credentials | CreateAccessKey, CreateLoginProfile | CreateServiceAccountKey | Credentials added to an app registration, in the Entra audit log |
| New identities | CreateUser, CreateRole | CreateServiceAccount | New service principals and users |
| Changed trust or permissions | UpdateAssumeRolePolicy, AttachRolePolicy, PutRolePolicy | SetIamPolicy | roleAssignments/write in the Activity Log |
| Borrowed identities | AssumeRole chains from the compromised principal | GenerateAccessToken, which is a Data Access log and off by default | Sign-ins by service principals |
The Google Cloud row about borrowed identities is worth reading twice. Service account impersonation is recorded only if Data Access logs are on for the IAM credentials API. Without them, you cannot tell who acted as which service account.
Each cloud has its own page with its audit log defaults, the fields that name an identity, and the first moves: AWS, Azure, Google Cloud, and Oracle Cloud. The log source planner lists which of these logs are off by default on your platforms, and how long each is kept.
Stolen credentials used to run AI models are a case of their own, with a bill and prompts to size. Stolen credentials running AI models works one from alert to containment.
Containment, by what was compromised
| Compromised | Contain it by | The catch |
|---|---|---|
| An AWS access key | Deactivating the key, then attaching a deny-all policy to its user. | Anything the key created, such as another key or a role, keeps working. |
| An AWS role's sessions | Revoke active sessions on the role. It adds a policy denying every credential issued before now. | Without it, stolen session credentials work until they expire, up to the role's maximum session length. |
| A Google Cloud service account | Disabling its keys, or the account itself, and removing who may impersonate it. | Disabling the account breaks every workload using it. Decide that knowingly. |
| An EC2 instance | Termination protection, snapshots, memory, then a quarantine security group. The evidence template does all of it. | Open connections can survive a security group change. Credentials taken from the instance work elsewhere until you revoke the role's older sessions. |
| A Compute Engine VM | A deny-all firewall rule matching a network tag, then adding the tag. Snapshot the disks first. | Changing a VM's service account means stopping it, so remove that account's permissions instead. |
| A Kubernetes pod | Relabelling it out of its controller, a deny-all network policy on a quarantine label, and cordoning the node. | Network policies only work if the cluster's network plugin enforces them. Kubernetes for defenders has the detail. |
Preserve, even when containment is urgent
Capture in order of volatility: what the machine looks like right now, then memory, then disk, then logs. One exception makes the order practical in the cloud. An EBS snapshot is fixed at the moment it is requested, so requesting it first costs a few seconds and keeps the disk clean of the memory image written after it.
Then keep what you captured trustworthy. Hash every file as it is collected, store it where nobody can change or delete it, and record who collected what and when. Object Lock and immutability policies do the storage part. The evidence template writes the hashes and a manifest for you.
Logs are evidence too. Export anything whose retention could run out during the investigation, and place a legal hold before anyone asks for one.
Who is in command
One person commands the incident, and that person does not also do the hands-on work. Commanding is a full-time job during the first hours. The usual roles:
- Incident commander. Decides, sets priorities, and owns the timeline.
- Operations lead. Runs the hands-on response and reports back.
- Communications. Writes the updates for leadership and anyone affected.
- Scribe. Keeps the decision log: what was decided, by whom, when, and on what evidence.
Agree before any incident which containment actions the commander may take without asking anyone, such as isolating a production instance or disabling a key. A commander who has to escalate each decision is slower than the attacker.
Running a cloud incident as an organisation covers the roles, the rhythm, handover, and disclosure clocks in full.
Update leadership on a fixed schedule; every hour during active containment is common. Use the same four parts every time: what we know, what we are doing, what we need, and when the next update is.
After: a review that changes something
A review is finished when each finding has become a change, owned by the team that owns the system. Many common findings map to a template here:
| Finding | The change |
|---|---|
| The attacker deleted or disabled the logs | Audit logs nobody can delete |
| Evidence was lost when the instance was replaced | Evidence that survives termination |
| A long-lived key leaked from a pipeline | Cloud access without stored keys |
| A privileged pod made escape easy | Stop risky pods before they start |
| Data was copied to an outside account | Data that cannot leave your organisation |
Practise before you need it
Run the evidence template against a test instance every quarter, and run a tabletop exercise with the people who would be on the call. The first time a procedure runs should never be the real incident.