Home/For responders/Host forensics, in and out of the cloud
Host forensics, in and out of the cloud
One-page refresher: Linux host triage One-page refresher: macOS host triage One-page refresher: Windows host triage
Cloud machines are still Linux, Windows, or macOS underneath. When the cloud audit log shows what an attacker did to the account, the host shows what they did on the machine: what ran, what stayed behind, and what it talked to.
Before you touch it
- Do not reboot or shut it down. Memory, network connections, and anything running from a deleted file are gone the moment it stops.
- Bring your own tools. Run trusted static binaries from your own media or a mounted evidence volume. The machine's own tools may have been replaced.
- Write down every command and the time. Anything you run changes the machine a little, and the record says what you changed.
- Hash what you collect as you collect it, and keep it where it cannot be changed, such as the locked bucket from the evidence template.
- In the cloud, snapshot the disk before capturing memory. Writing a memory image changes the disk; the snapshot is fixed at the moment you ask for it.
Linux
Memory and running state
AVML captures memory without a kernel module and writes it in LiME format.
Volatility 3 then needs a symbol table built for that exact kernel, so record
uname -r before you start and keep the matching debug symbols.
| Question | Command |
|---|---|
| What is running, and since when | ps -eo pid,ppid,user,lstart,args |
| What it is talking to | ss -tunap |
| Programs running from a deleted file | ls -l /proc/*/exe 2>/dev/null | grep deleted |
| Deleted files still held open | lsof -nP +L1 |
| Loaded kernel modules | lsmod |
Where persistence hides
| Place | Look at |
|---|---|
| Scheduled jobs | /etc/crontab, /etc/cron.*, /var/spool/cron/, and systemctl list-timers --all |
| Services | /etc/systemd/system/, /usr/lib/systemd/system/, and each user's ~/.config/systemd/user/ |
| SSH keys | Every authorized_keys, including any file AuthorizedKeysFile names in sshd -T |
| Library preloading | /etc/ld.so.preload, which loads a library into every program. It is usually empty or missing |
| Shell start-up | /etc/profile.d/, ~/.bashrc, ~/.profile |
| Accounts | New entries in /etc/passwd, and /etc/sudoers.d/ |
| Sign-in modules | Unfamiliar files or lines in /etc/pam.d/ |
Logs
journalctl --since for the journal, /var/log/auth.log or
/var/log/secure for sign-ins, and last and lastb for
successful and failed logins. auditd, where it runs, records far more; the
audit rules template sets it up
for the questions below. The
host log template keeps a copy
off the machine, which matters because attackers clear these first.
What the cloud adds
- User data and cloud-init. An attacker who can change an
instance's user data gets a script run at the next boot. Read
/var/lib/cloud/instance/user-data.txtand/var/log/cloud-init.log. - Agents. The Systems Manager agent logs to
/var/log/amazon/ssm/. Commands run through it appear there and in the cloud audit log. - The instance's credentials. Reading them from the metadata service leaves nothing on the host. Look in the cloud audit log for the instance's role used from an address that is not the instance.
macOS
Memory and running state
Full memory capture on current Macs is not practical for most teams, because System Integrity Protection keeps tools out of the kernel. Put the effort into running state and the unified log, and collect both early.
| Question | Command |
|---|---|
| What is running | ps -axo pid,ppid,user,lstart,command |
| What it is talking to | lsof -nP -i |
| Capture the unified log | sudo log collect --last 3d --output host.logarchive |
The unified log often keeps only days of history on a busy Mac, so capture it before anything else that can wait.
Where persistence hides
| Place | Look at |
|---|---|
| Launch agents and daemons | ~/Library/LaunchAgents/, /Library/LaunchAgents/, /Library/LaunchDaemons/. Read each property list's ProgramArguments |
| Login items and background tasks | sudo sfltool dumpbtm on macOS 13 and later lists everything registered to start |
| Configuration profiles | sudo profiles list. A profile nobody in IT installed can change proxies, certificates, and more |
| Extensions | systemextensionsctl list |
| Shell start-up | ~/.zshrc, ~/.zshenv, /etc/zshrc |
| Sign-in plug-ins | /Library/Security/SecurityAgentPlugins/ |
Artifacts that answer specific questions
- Where did this file come from?
xattr -p com.apple.quarantineon the file names the app that downloaded it. The database at~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2keeps the download address. - What was granted access to the disk, camera, or screen? The
privacy database, TCC, at
~/Library/Application Support/com.apple.TCC/TCC.dband/Library/Application Support/com.apple.TCC/TCC.db. Reading the system one needs Full Disk Access for your terminal. - Did Gatekeeper let it run? Search the unified log for the
syspolicydprocess around the time the file first ran.
In a managed fleet, the device management service can push a collection script and isolate the Mac, which is faster than reaching it by hand.
Windows
Memory and running state
Capture memory first, with a tool such as WinPmem. Volatility 3 fetches the
Windows symbols it needs from Microsoft, so an offline workstation needs them
cached in advance. Then record running state:
Get-Process | Select Id,ProcessName,Path,StartTime and
Get-NetTCPConnection -State Established.
Where persistence hides
| Place | Look at |
|---|---|
| Run keys | HKLM and HKCU\Software\Microsoft\Windows\CurrentVersion\Run and RunOnce |
| Scheduled tasks | schtasks /query /fo LIST /v, and the files under C:\Windows\System32\Tasks |
| Services | HKLM\SYSTEM\CurrentControlSet\Services, and event 7045 in the System log for each new one |
| WMI subscriptions | Get-CimInstance -Namespace root/subscription -ClassName __FilterToConsumerBinding. These run code on an event, with no file in a startup folder |
| Startup folders | Each user's and the all-users Startup folder |
Sysinternals Autoruns reads nearly all of these in one pass:
autorunsc -a * -c -h -s lists every entry as CSV, with hashes and
signature checks.
Events worth knowing by number
| Event | Log | Means |
|---|---|---|
| 4624, 4625 | Security | A sign-in, and a failed one. Logon type 10 is Remote Desktop, type 3 is over the network |
| 4672 | Security | A sign-in given administrator-level privileges |
| 4688 | Security | A process started. Its command line is recorded only if that policy is switched on, which the Windows host log template does |
| 4720, 4732 | Security | An account created, and one added to a group |
| 4698 | Security | A scheduled task created |
| 1102 | Security | The security log was cleared |
| 7045 | System | A service installed |
| 4104 | PowerShell Operational | A PowerShell script block, recorded when script block logging is on |
Evidence that something ran
Prefetch files in C:\Windows\Prefetch record programs run and when,
but are often switched off on servers. Amcache and the SRUM database also record
execution. ShimCache lists programs the system has seen; on current Windows, an
entry there alone does not prove the program ran. KAPE collects all of these in
one pass, and Eric Zimmerman's tools parse them.
Sysmon
Sysmon records what the built-in logs leave out: each process start with its hash and parent, outbound connections, driver loads, and changes to autostart registry keys. The host log template installs it with a configuration that names every event type, and forwards it off the machine with Windows Event Forwarding.
What the cloud adds
The Systems Manager agent logs to C:\ProgramData\Amazon\SSM\Logs, and
the Azure guest agent to C:\WindowsAzure\Logs. As on Linux, the
instance's credentials leave no trace on the host when read; the cloud audit log
is where their misuse shows.
On every host
- Build one timeline. Plaso's
log2timelineputs file times, logs, and registry changes on one clock. Pair it with the cloud audit log for the same hours. - Ask with osquery. Where the
osquery template runs, its results log
already contains what changed before anyone looked, and
osqueryianswers the same questions on the spot on Linux, macOS, and Windows. - Compare with a clean machine. A machine built from the same image shows what should be there, which makes the extra entries stand out.
- Analyse on an isolated machine. The forensic workstation reads evidence and cannot reach anything live, so a sample that tries to call home is refused.
The Linux, macOS, and Windows refreshers put each section on one printable page.