Home/For responders/Incident response on Google Cloud
Incident response on Google Cloud
Google Cloud records every change to your projects by default and almost none of the reads. Most of an investigation depends on which of those two facts you prepared for.
What Google Cloud records, and what it leaves out
| Audit log | On by default | Kept |
|---|---|---|
| Admin Activity | Always. It cannot be switched off | 400 days in the _Required bucket, at no charge |
| System Event | Always | 400 days in _Required |
| Data Access | Off, except for BigQuery | 30 days in _Default unless you change it, and charged |
| Policy Denied | On, and charged for storage | 30 days in _Default |
Admin Activity answers "who changed this": new keys, new role grants, new firewall rules, new VMs. Data Access answers "who read this": objects downloaded from a bucket, secrets fetched, tables queried. With Data Access off, a data theft through legitimate read calls leaves no record in the audit log. Switch it on for the services that contain sensitive data, at the organisation level, before you need it, and keep the logs longer than 30 days. The audit log template copies them where nobody in the project can delete them.
Reading an identity out of the logs
Each entry names the caller in protoPayload.authenticationInfo.principalEmail.
Three patterns need more than that field:
- A service account key. Creation is
google.iam.admin.v1.CreateServiceAccountKeyin Admin Activity. Calls made with a key carryserviceAccountKeyName, which names the exact key, so you can tell a stolen key from the service's normal use. Compute Engine and App Engine do not log the key name. - Impersonation. The token request is
GenerateAccessToken, a Data Access log of the IAM API, so it is only recorded if you switched those on. Calls made with the token carryserviceAccountDelegationInfo, which names who asked. Follow that chain back to a person. - A workload identity. A pod or a pipeline acting as a service account is still that service account in the logs. The workload identity template limits which workloads can do it.
gcloud logging read 'protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"' --freshness=30d --project <project>
gcloud logging read 'protoPayload.authenticationInfo.serviceAccountKeyName:"<key id>"' --freshness=30d --project <project>
gcloud logging read 'protoPayload.authenticationInfo.principalEmail="<identity>"' --freshness=7d --project <project>
Containment, by what was compromised
The containment refresher puts the first move for each kind of compromise on one printable page.
| Compromised | First move | Watch out |
|---|---|---|
| A service account key | gcloud iam service-accounts keys disable the key. Delete it once the investigation no longer needs it | Other keys on the same account. List them all |
| A user account | Suspend it in Cloud Identity or Workspace, and revoke its sessions there | Keys and service accounts it created, which outlive the suspension |
| A VM | Snapshot its disks, then a deny-all firewall rule that targets it by service account or network tag | Firewall changes apply to new connections. Existing ones persist while traffic keeps flowing |
| A workload identity | Remove the binding that lets the workload act as the service account | Tokens already issued keep working until they expire |
Target isolation rules by service account when you can. Anyone who can edit the VM can change its network tags, and with them which rules apply. The evidence template does the snapshot and the isolation in that order when someone labels the VM.
Saved queries
20 saved queries from three templates open in this cloud's console, named as the console lists them. The Standard and Hardened levels install them; the Quick start installs none. They use the Logging query language, apart from one in SQL for Log Analytics.
| Saved query | What it answers | Template that installs it | Where to open it |
|---|---|---|---|
AI models: logging-weakened | The events the logging-weakened alert matches. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: guardrails-weakened | The events the guardrails-weakened alert matches. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: ai-api-enabled | The events the ai-api-enabled alert matches. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: model-recon | The events the model-recon alert matches. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: one principal's model calls | Model calls by one principal. Replace the email first. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: calls made with a service account key | Model calls signed with a service account key. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: unexpected-caller | Model calls by anyone outside the callers you listed. Only when model_callers is set. | Know who is using your AI models | Logs Explorer, Saved queries, Shared, in the security project |
AI models: calls by principal and model, last seven days | Who called which model, and how often, over the last seven days. | Know who is using your AI models | Log Analytics, Saved queries, in the security project |
Insider: changes by person | Every change that alerts, with who made it. | Notice data walking out the door | Logs Explorer, Saved queries, Shared, in the security project |
Insider: permission changes | Who was given new access. Run it first when an alert names a person. | Notice data walking out the door | Logs Explorer, Saved queries, Shared, in the security project |
Insider: one principal | Everything one principal did. Replace the email first. | Notice data walking out the door | Logs Explorer, Saved queries, Shared, in the security project |
Insider: object reads | Object reads in the sensitive projects. Only when sensitive_projects is set. | Notice data walking out the door | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: admin-granted | The events the admin-granted alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: sign-in-changed | The events the sign-in-changed alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: two-step-off | The events the two-step-off alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: password-reset-by-admin | The events the password-reset-by-admin alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: account-warning | The events the account-warning alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: mail-forwarded-out | The events the mail-forwarded-out alert matches. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: one user | Everything one user did, or that names them. Replace the email first. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Workspace: failed sign-ins | Failed Workspace sign-ins. | Catch account takeover at the identity provider Google Workspace | Logs Explorer, Saved queries, Shared, in the security project |
Forensics on a Compute Engine VM
- Disks. A snapshot is crash-consistent without pausing the VM, the same as pulling the power: writes still in memory may be missing. A machine image also captures the VM's configuration, metadata, and every disk together.
- Memory. Google Cloud documents no way to export a VM's memory without software inside it. Capture it with a tool such as AVML inside the guest, after the snapshot, written somewhere other than the disks under investigation.
- Serial port output. The console and API return only the last 1 MB per port. Turn on serial port logging to Cloud Logging beforehand if you want more.
- Where to look at it. Attach disks made from the snapshots to an isolated workstation, read-only. The forensic workstation template builds one with no internet route and DNS limited to Google's APIs.
Detection you can switch on
Security Command Center's Standard tier has no threat detection services. Premium and Enterprise add three:
| Service | Watches |
|---|---|
| Event Threat Detection | The audit logs and other log streams, for patterns such as an anomalous role grant or service account impersonation |
| Container Threat Detection | GKE containers: added binaries run, reverse shells, container escapes |
| VM Threat Detection | VM memory, read from the hypervisor with no agent, for crypto mining and kernel rootkits. Not available on Confidential VMs or Arm VMs. Disks under your own key are not scanned for malware, though memory still is |
Google has announced that the Enterprise tier ends on May 21, 2027. The compliance template routes findings to your alert channel, and the SIEM template sends the logs themselves.
Containers on Google Cloud
GKE adds a cluster audit log, nodes that autoscaling and repair can replace, and on Autopilot, nodes you cannot reach at all. Containers on GKE and EKS covers each of these, and the container evidence template captures a compromised pod.