rsyslog Central Log Server: Receive and Forward Logs

When you manage more than a handful of Linux servers, switches and firewalls, ssh-ing into each box to read logs stops working - and when a server dies, its logs die with it. An rsyslog central log server collects syslog from every device into per-host directories, giving you one searchable place for troubleshooting and audits. rsyslog is pre-installed on most distributions, so the whole project is two small config files: one on the collector to receive, one on each client to forward. This guide covers both sides with Ubuntu commands.

Why Centralize Logs at All

Logs stored only on the machine that generated them are nearly useless the moment that machine is the problem. A host that fills its disk, gets compromised, or simply reboots loses the very evidence you need. Centralizing fixes three things at once: survivability (the evidence lives somewhere else), correlation (you can line up an SSH failure on one host with a firewall drop on another using timestamps), and retention (one place to enforce a retention policy instead of fifty).

Systemd has replaced traditional syslog for local journaling, but the syslog protocol is still what nearly every network device, appliance and daemon speaks. rsyslog reads the journal, understands the syslog protocol natively, and can therefore act as the protocol-agnostic landing zone for everything from Cisco switches to appliance firmware. That is why it remains the pragmatic default for a self-hosted log server.

How rsyslog Selects Logs: Facilities and Severities

Before touching config, be fluent in the two fields every syslog message carries, because all selective forwarding is built on them:

• Facility — the subsystem that produced the message: auth and authpriv (login and privilege events), kern (kernel), mail, cron, daemon, local0–local7 (free for your own applications), and so on.
• Severity — the importance, from emerg (0) through alert, crit, err, warning, notice, info down to debug (7).

A selector such as auth,authpriv.* means “anything from those two facilities at any severity,” while *.err means “severity error and above from all facilities.” Getting these right is what keeps a busy collector from drowning in debug spam.

Step 1: Configure the Collector (Server Side)

Create /etc/rsyslog.d/10-remote-logs.conf on the log server to load the UDP and TCP input modules and organize remote logs per hostname and program:

module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")

$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
:source, !isequal, "localhost" -?RemoteLogs
:source, !isequal, "localhost" stop

Then restart and open the firewall port:

sudo systemctl restart rsyslog
sudo ufw allow 514/tcp
sudo ufw allow 514/udp

UDP is the traditional syslog transport but can lose messages under load; TCP guarantees delivery and is the recommended choice for production. rsyslog's modern RainerScript syntax can also write this with action(type="omfile" dynaFile="RemoteLogs").

The two lines after the template are worth reading carefully, because they are what keeps the collector tidy. The first, :source, !isequal, "localhost" -?RemoteLogs, says: if the message did not originate on this host, write it through the RemoteLogs dynamic-file template. The leading dash means “do not complain if the file cannot be created right now,” and the question mark enables the dynamic filename. The second line, stop, ends processing for that message so it does not also land in the local /var/log/syslog. Without the stop you get double entries and a rapidly growing local file.

Step 2: Forward Logs from Clients

On every client create /etc/rsyslog.d/50-forward-logs.conf. A single @ means UDP, @@ means TCP:

# Forward everything over TCP to the central server
*.* @@192.168.1.100:514

Selective forwarding works with facility/severity selectors or property filters:

auth,authpriv.* @@log-server.example.com:514   # only auth logs
*.err @@log-server.example.com:514             # errors and above
if $programname == 'nginx' then @@log-server.example.com:514

sudo systemctl restart rsyslog

Note the difference between the two syntaxes: the leading selector form (auth,authpriv.*) is the classic filter syntax and works on any rsyslog version, while the if ... then form is RainerScript and gives you access to every property — program name, hostname, message text, structured data. Use the classic form for simple cases and RainerScript when the decision needs logic.

Step 3: Test the Pipeline

Send a test message from a client and confirm it lands on the collector:

logger -p local0.info -t shipdemo "central pipeline test"
# on the collector:
sudo grep "central pipeline test" /var/log/remote/*/*.log

Also validate the collector config before restarting with sudo rsyslogd -N1 and confirm the listener with sudo ss -tlnp | grep 514.

Two details make this test reliable. First, use the -t shipdemo tag so the message is easy to spot in a file whose name is derived from the program name — with the tag set, look for a file literally called shipdemo.log. Second, if nothing appears, check the collector received anything at all with sudo tcpdump -i any -n port 514: seeing packets arrive but no file means the rsyslog rule or template is wrong; seeing no packets means the client, the network or a firewall is blocking the path.

Hardening the Collector

A log server that accepts input from the network is an attack surface, and one that stores unbounded data is a disk-space incident waiting to happen. A few changes are worth making on day one:

• Always validate before restart. Run sudo rsyslogd -N1 after every edit. A single typo causes rsyslog to refuse to start, and on some systems the failure is silent enough that you only notice when logs stop arriving.
• Give /var/log/remote its own filesystem or quota. A chatty device in a loop can fill the root filesystem and take down the collector. A dedicated volume, or a quota on /var/log, contains the blast radius.
• Restrict who can read the logs. Remote logs frequently contain credentials in URLs or tokens in debug output. Tighten permissions on /var/log/remote (for example 0750 owned by root:adm) and be careful about exposing the directory through any web tool.
• Bind the listener if the collector has multiple interfaces. Adding Address="10.0.0.5" to the input() lines limits reception to the management network instead of every interface.

Reliable Delivery with the omfwd Queue

The single-line *.* @@server rule is fine for a lab, but on a production client it has a flaw: if the collector is briefly unreachable — a restart, a network blip — the messages sent during that window are dropped. The fix is an explicit forwarding action with an on-disk queue that buffers while the destination is down:

action(type="omfwd"
       target="log-server.example.com" port="514" protocol="tcp"
       queue.type="LinkedList"
       queue.filename="fwdq"
       queue.maxdiskspace="2g"
       queue.saveonshutdown="on"
       action.resumeRetryCount="-1")

The key parameters: queue.type="LinkedList" creates an in-memory queue, queue.filename makes it spill to disk (which is what turns it into a persistent queue), action.resumeRetryCount="-1" means “keep retrying forever,” and queue.saveonshutdown="on" persists the buffer across a reboot. With this in place a collector outage causes a delay, not a gap.

Retention and Rotation

Centralized logs grow fast, so rotation is not optional. Because the files are created dynamically by hostname and program, a wildcard logrotate stanza handles them all:

/var/log/remote/*/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root adm
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Drop this in /etc/logrotate.d/remote-logs. The postrotate call tells rsyslog to reopen its file handles after rotation — without it, the daemon keeps writing to the rotated file and the new one stays empty. Thirty compressed days is a reasonable default; align the number with whatever your audit or compliance policy requires.

TLS-Encrypted Remote Logging

Plain syslog sends everything in cleartext, including the credentials that appear in auth logs. If the collector is reachable across any untrusted segment, encrypt the transport. rsyslog can do this natively with the gtls driver once you have a certificate authority and per-host certificates:

# On the collector: accept TLS on 6514
module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1"
       StreamDriver.AuthMode="x509/name")
input(type="imtcp" port="6514" ruleset="remote")

# On a client: forward over TLS
action(type="omfwd" target="log-server.example.com" port="6514"
       protocol="tcp"
       StreamDriver.Name="gtls" StreamDriver.Mode="1"
       StreamDriver.AuthMode="x509/name"
       StreamDriverPermittedPeers="log-server.example.com")

With StreamDriver.AuthMode="x509/name", the collector validates that the client certificate’s subject matches the peer it claims to be — this also stops one compromised host from impersonating another in the logs. Generate a private CA, issue a certificate per sender, and you have authenticated, encrypted, tamper-evident log transport.

Hints for Network Devices

The same collector accepts logs from switches and firewalls: point their syslog server setting at the collector IP and they appear under their hostname in /var/log/remote/. Add logrotate for /var/log/remote/*/*.log - centralized logs grow fast.

Two device-side details matter. First, set the source-interface so the device’s management IP is used and the hostname it reports is stable; otherwise logs end up under an unhelpful name. Second, align the logging severity and timestamp settings on both ends — a Cisco device configured with logging trap informational and service timestamps log datetime localtime produces entries that are both useful and easy to correlate:

logging host 192.168.1.100 transport tcp port 514
logging trap informational
logging source-interface Vlan10
service timestamps log datetime msec localtime show-timezone

Also confirm the devices’ clocks: correlating logs across many senders is hopeless without synchronized time. See the related guides below for NTP troubleshooting.

Monitoring and Alerting on the Central Store

A silent log server is itself a failure — you will not notice that forwarding broke until you need the logs. A few cheap checks close that gap: alert when /var/log/remote stops growing, when the collector’s listening socket disappears, or when the disk holding /var/log/remote crosses a threshold. If you outgrow per-file searching, the natural next step is a purpose-built aggregator — see the Grafana Loki guide below for indexing and querying without maintaining Elasticsearch.

Related Guides on This Site

Keep the time on all those devices consistent with chrony NTP sync and drift troubleshooting, and for monitoring stack alternatives see Prometheus snmp_exporter monitoring. To take the next step from flat files to a queryable log store, read Grafana Loki: log aggregation without the index bill, and for the device side of syslog configuration see Cisco IOS logging and syslog config: levels, buffer and traps.

原文链接：https://www.redhat.com/en/blog/log-aggregation-rsyslog