Logs and a disk that fills up

Find out what is eating the disk

Start at the top and work down:

df -h — which filesystem is full

du -xh --max-depth=1 / | sort -h | tail -15 — then repeat inside the biggest directory

journalctl --disk-usage — how much the journal is holding

One trap worth knowing: if you deleted a large log file but the space did not come back, a process still has it open. Find it with lsof +L1 and restart that service. Truncating with : > /var/log/big.log avoids the problem entirely — never rm a log that something is writing to.

logrotate

Most packages install their own rules in /etc/logrotate.d/. Your own application needs one too:

/var/log/myapp/*.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  copytruncate
}

copytruncate copies the file and then empties the original, so a process that holds the file open keeps writing to the same handle. Without it, and without a postrotate that signals your application, rotation produces an empty new file that nothing ever writes to.

Test before trusting: logrotate -d /etc/logrotate.d/myapp shows what it would do without doing it.

Cap the journal

systemd's journal grows until it is told not to. In /etc/systemd/journald.conf:

SystemMaxUse=2G
MaxRetentionSec=1month

Then systemctl restart systemd-journald. To reclaim space now:

journalctl --vacuum-size=1G

How long to keep what

Web access logs

Two to four weeks is plenty for debugging. If you keep them longer, remember they contain IP addresses, which are personal data — see who is responsible under the GDPR.

Application errors

Longer, and somewhere searchable. These are the ones you actually go back to.

Authentication logs

Keep them. When something goes wrong, auth.log is the first file anybody asks for.

Debug logging

Off in production, or switched on deliberately and switched off again. It is the usual culprit behind a disk that filled in one night.

Alert before it is full

A disk alert at 80% turns an outage into a chore. Put it in your monitoring — see monitoring your server — and include inodes as well as bytes. A filesystem with millions of tiny session files can run out of inodes while df -h still shows plenty of space; df -i is the one that tells you.

---

Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base