iac.htora.dev · security templates

Home/Templates/Send host logs off the machine

Send host logs off the machine

Forward system logs to a collector so they outlive the machine that wrote them.

Ansible

Why bother

Logs that only exist on the host disappear when the host does, which is exactly what someone arranges when they want the story to go away. Sending them somewhere else as they are written means you still have them afterwards. It also means one search covers the whole fleet.

How you know it worked

Make something happen on a host, such as a login or a new process. Find it on the collector within a minute or two.

How it works

1Something happens on the hosta login, a new process, a service install2Recorded locallyjournald and rsyslog on Linux, event logs and Sysmon on Windows3Held until senta disk queue on Linux, the event log itself on Windows4The collectorover TLS from Linux, Kerberos-encrypted WinRM from Windows
Step by step, from the thing that happens to the result you want.

Set it up

Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.

Ansible. rsyslog forwarding, with a queue that survives the collector being down

What you need first

  • A collector reachable from the hosts that accepts syslog over TCP.
  • For the Standard version, the CA that signed the collector certificate present on every host. The role checks and stops if it is missing.

What it creates

  • /etc/rsyslog.d/10-forward.conf with a disk-backed queue
  • A spool directory for messages waiting to be sent
  • A raised RateLimitBurst in journald.conf

The code

Quick startReadyone file, 63 lines
One playbook. Plain TCP to a collector, with the disk queue. Use it on a network you control.
Change these first
  • collector_host: logs.example.internal playbook.yml, line 23
  • collector_port: 514 playbook.yml, line 24
Run it
  1. ansible all -i inventory.ini -m ping
    Proves Ansible can reach every machine before anything changes.
  2. ansible-playbook -i inventory.ini playbook.yml --check --diff
    Shows each change it would make, line by line. Changes nothing.
  3. ansible-playbook -i inventory.ini playbook.yml
    Makes the change.
  4. ansible-playbook -i inventory.ini playbook.yml
    Run it once more. The recap should end with changed=0.

No inventory yet? A one-machine inventory.ini:

[all]
web-01 ansible_host=10.0.0.11 ansible_user=ubuntu
Download playbook.yml
Checks

Syntax passedAnsible lint passed

Last run 2026-10-02.

outcomes/host-log-shipping/linux/t0
playbook.yml
Open playbook.yml as plain text
StandardReadyfive files, 176 lines
A role. Adds TLS with peer name checking, a CA presence check that stops the run, the journald limit, and a test message at the end.
Run it
  1. ansible all -i inventory.ini -m ping
    Proves Ansible can reach every machine before anything changes.
  2. ansible-playbook -i inventory.ini site.yml --check --diff
    Shows each change it would make, line by line. Changes nothing.
  3. ansible-playbook -i inventory.ini site.yml
    Makes the change.
  4. ansible-playbook -i inventory.ini site.yml
    Run it once more. The recap should end with changed=0.

No inventory yet? A one-machine inventory.ini:

[all]
web-01 ansible_host=10.0.0.11 ansible_user=ubuntu
Download all five files as a zip
Checks

Syntax passedAnsible lint passed

Last run 2026-10-02.

outcomes/host-log-shipping/linux/t1
roles/log_forwarding/defaults/main.yml
Open main.yml as plain text
roles/log_forwarding/handlers/main.yml
Open main.yml as plain text
roles/log_forwarding/tasks/main.yml
Open main.yml as plain text
roles/log_forwarding/templates/forward.conf.j2
Open forward.conf.j2 as plain text
site.yml
Open site.yml as plain text
HardenedReadythree files, 145 lines
Standard's roles, unchanged, with your settings in settings.yml. Rules refuse settings that turn TLS off, weaken peer checks, drop the disk queue, turn rate limiting on, or skip the delivery check. A daily check-mode run reports drift, and ansible-lint runs on the production profile.
Run it
  1. ansible all -i inventory.ini -m ping
    Proves Ansible can reach every machine before anything changes.
  2. ansible-playbook -i inventory.ini site.yml --check --diff
    Shows each change it would make, line by line. Changes nothing.
  3. ansible-playbook -i inventory.ini site.yml
    Makes the change.
  4. ansible-playbook -i inventory.ini site.yml
    Run it once more. The recap should end with changed=0.

No inventory yet? A one-machine inventory.ini:

[all]
web-01 ansible_host=10.0.0.11 ansible_user=ubuntu
Download all three files as a zip
Checks

Syntax passedAnsible lint passed

Last run 2026-10-02.

outcomes/host-log-shipping/linux/t2
guard.yml
Open guard.yml as plain text
settings.yml
Open settings.yml as plain text
site.yml
Open site.yml as plain text

How to check it worked

Write a test line with logger -t iac-test hello. Find it on the collector within a few seconds.

The part that catches people

journald throws messages away above its own rate limit before rsyslog ever sees them. Raising the rsyslog limit alone changes nothing, which is why a burst during an incident still goes missing. The Standard role raises both.

How to undo it

Remove the file and restart rsyslog. Anything already queued on disk is either delivered or dropped, depending on how long the collector has been away.

What it costs

None on the host beyond the disk the queue uses. The charge, if there is one, is on the receiving side per gigabyte.

Registry 0.23.0. Built 2026-11-17.

Made by Habibullah Tora. Source on GitHub. Code under the MIT licence, writing under CC BY 4.0.