systemd journald Persistent Logging Configuration

The most frustrating log collection bug is the one you notice after a reboot: the journal is empty, and the evidence of what happened before the restart is gone. That is by design on most distributions — systemd-journald keeps logs in a memory-backed ring buffer under /run/log/journal unless you explicitly ask for persistent storage. This article shows both ways to make journald persistent, how to verify it actually worked, and how to keep it from silently growing past its size limit.

Volatile versus persistent storage

The journal stores data either persistently under /var/log/journal or volatilely under /run/log/journal. By default, data is persistent only if /var/log/journal exists at boot; otherwise journald falls back to volatile storage, and everything is lost when /run is cleared. The Storage= setting in journald.conf makes that decision explicit rather than dependent on a directory being present.

Method 1: create the directory

# create the persistent journal directory
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

# RHEL 9/10 and other recent releases: flush the volatile journal to disk
journalctl --flush

On older releases the directory alone is enough — the next loggable event starts writing to disk. On current releases run journalctl --flush as well so the data already buffered in /run is moved to /var instead of waiting for the next boot.

Method 2: set Storage=persistent

sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
systemctl restart systemd-journald.service
journalctl --flush

With Storage=persistent, data is written preferably to disk under /var/log/journal (created if needed) with a fallback to /run/log/journal during early boot or when the disk is not writable. That fallback is what makes persistent safer than volatile for servers that mount /var late. Check the effective setting afterwards:

cat /etc/systemd/journald.conf
systemctl status systemd-journald
journalctl --disk-usage
ls -l /var/log/journal/

Permissions and ownership

If you create the directory by hand, fix the access modes so the journal is readable by the intended groups. The canonical recipe from the upstream documentation:

setfacl -Rnm g:wheel:rx,d:g:wheel:rx,g:adm:rx,d:g:adm:rx /var/log/journal/

Getting this wrong produces a journal that exists on disk but is empty because journald refuses to write into a directory the service user cannot access — check systemctl status systemd-journald for permission errors before assuming the configuration is wrong.

Size limits, rotation and vacuum

# inspect usage
journalctl --disk-usage

# free space down to a target size
journalctl --vacuum-size=2G
journalctl --vacuum-time=30d

# force rotation now
journalctl --rotate

A long-standing operational surprise: setting SystemMaxUse= or RuntimeMaxUse= does not guarantee the journal stays under that number. Persistent journals are rotated into archived files, and vacuuming only frees whole archived files, so a journal can sit above the limit while journalctl --vacuum-size reports "freed 0B". The practical fix is to rotate first, then vacuum:

journalctl --rotate
journalctl --vacuum-size=2G
journalctl --disk-usage

Useful query patterns

journalctl -b              # current boot only
journalctl -b -1           # previous boot (needs persistence)
journalctl -k              # kernel messages only
journalctl -u sshd --since "1 hour ago"
journalctl -p err -n 200
journalctl _HOSTNAME=web01 -S today

journalctl -b -1 is the command that proves persistence is working: if it returns "Failed to look up boot -1: No such boot ID", the journal is still volatile no matter what Storage= says.

Checklist for a new server build

• Enable persistence in your image template — not on the day something crashes.
• Set both SystemMaxUse and SystemKeepFree so the journal can never fill the root filesystem.
• Forward to a central collector as well; a journal on the host is not a substitute for remote retention.
• Confirm with journalctl -b -1 after the first reboot, before you need the data.

See also: building a central rsyslog server, systemd timers vs cron, and cleaning up when a disk fills.

原文链接：https://access.redhat.com/solutions/696893