Shipping logs off the machine

Two reasons, not one

• Trust. Anyone with root can rewrite auth.log. A copy that left the machine within a second cannot be edited retroactively — see a compromised server.
• Correlation. With five machines, the interesting question is almost always “what happened across all of them at 14:32”, and that is impossible with five separate journalctl sessions.

Both depend on the clocks agreeing — see keeping the clock right.

The simple version: rsyslog

Already installed nearly everywhere. On each client, in /etc/rsyslog.d/90-remote.conf:

*.* @@10.0.0.9:514

Two @ means TCP, which actually matters — UDP silently drops messages under load, which is precisely when you want them. Add a disk queue so nothing is lost if the collector is briefly away.

On the collector, enable the TCP input and write per-host files. Listen only on the private network — see connecting your own servers.

The searchable version: Loki

Promtail on each machine reads the journal and ships to Loki; you query it in Grafana next to your metrics. Loki indexes labels rather than full text, so it is cheap to run and fast enough for operational questions.

It pairs naturally with Prometheus and Grafana: an alert fires, you click through to the logs for that host and minute. Elasticsearch does full-text better but wants far more machine — see Elasticsearch and OpenSearch.

What to ship

Always

auth.log and sudo, systemd unit state, kernel messages, your application's errors.

Usually

Web access logs, mail logs, firewall drops.

Rarely

Debug output. It is enormous and it is the thing that fills the collector's disk.

Retention, with the GDPR in mind

Web and mail logs contain IP addresses, which are personal data. You need a retention period you can justify and that actually takes effect — not "we keep everything forever because storage is cheap".

Two to four weeks covers nearly all debugging. Keep authentication logs longer; those are what anybody asks for after an incident. Set it in the tool so it happens without anyone remembering. See who is responsible under the GDPR and logs and a disk that fills up.

Protect the collector

It now holds the most sensitive summary of your estate that exists. Private network only, its own firewall rules, its own backups, and a separate machine from anything it monitors. If you would not put your password manager on a box, do not put the log collector there either.

---

Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base