Linux Network Tuning with sysctl: TCP Buffers and Backlogs

Default Linux network settings are conservative because they have to fit a laptop, a container and a 400G database server equally well. On a busy proxy, an object store or a NAS, that conservatism shows up as dropped connections and truncated throughput. This guide tunes the parameters that matter in practice - TCP send and receive buffers, the listen backlog, connection-tracking and file-descriptor ceilings - with a repeatable method: measure, change one group at a time, re-measure with iperf3 and real workload traffic, and keep the change in a versioned file under /etc/sysctl.d/.

Start with evidence, not with numbers from the internet

ss -s
ss -lnt | head
nstat -az | grep -Ei 'drop|retrans|prune|overflow'
netstat -s | grep -i listen
dmesg | grep -i 'Possible SYN flooding|nf_conntrack' | tail
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem

SYNs to LISTEN sockets dropped and rising nstat overflow counters are the signals that justify backend and backlog tuning. If those counters are flat, changing sysctls will do nothing except make the next admin's life harder.

TCP buffers and the auto-tuning window

# /etc/sysctl.d/10-tcp-socket-buffers.conf
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 262144 67108864
net.ipv4.tcp_wmem = 4096 262144 67108864
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.netdev_max_backlog = 16384
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

sysctl -p /etc/sysctl.d/10-tcp-socket-buffers.conf
iperf3 -s                       # on the receiver
iperf3 -c 10.0.0.20 -t 20 -P 4  # on the sender, 4 parallel streams
iperf3 -c 10.0.0.20 -t 20 -u -b 10G

The three numbers in tcp_rmem and tcp_wmem are minimum, default and maximum. Auto-tuning moves the effective size between default and max as the connection grows, so raising only the third value takes effect dynamically; changing the second value requires restarting the applications to be picked up. Match rmem_max and wmem_max to the maximum you set for TCP, otherwise the socket buffer ceiling clips your auto-tuned value.

Backlog and accept queue

net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_abort_on_overflow = 0

somaxconn caps how many established-but-unaccepted connections can wait; an application can request a larger backlog, but the kernel clamps it to this value. Always raise the application's own setting too (nginx backlog, Gunicorn --backlog, Java acceptCount) or the kernel ceiling is not the binding constraint. Keep tcp_syncookies enabled - it is a safety net, not a substitute for a bigger backlog.

Connection tracking, ports and file descriptors

net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
fs.file-max = 2097152

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep -i 'table full'

A NAT or load-balancing host that hits nf_conntrack: table full, dropping packet drops traffic with no application log at all - always size the table against expected concurrent flows and watch the count. For file descriptors, also check the per-process limit (prlimit --nofile --pid <pid>) and the systemd unit's LimitNOFILE; raising fs.file-max alone changes nothing.

Validation and rollback

sysctl -a > /root/sysctl-before.txt
# apply new drop-in file, then
sysctl -a > /root/sysctl-after.txt
diff /root/sysctl-before.txt /root/sysctl-after.txt

# runtime rollback for a single key
sysctl -w net.ipv4.tcp_rmem="4096 87380 6291456"

Persist changes only in /etc/sysctl.d/*.conf (never edit /etc/sysctl.conf by hand across a fleet) and record the before/after dump next to the change ticket. A tuning change without a baseline measurement and a rollback path is a future three-in-the-morning incident. Useful companions: iperf3 testing guide, Linux network troubleshooting, ethtool NIC diagnostics and tc traffic shaping.

What each counter actually tells you

Tuning starts by knowing which number moves when the symptom appears. These are the counters worth watching, all available from nstat, ss or /proc/net/netstat:

• TcpExtListenDrops / "SYNs to LISTEN sockets dropped": the accept queue filled. This is the single most useful signal for backlog tuning, and it rises before users notice anything.
• TcpExtListenOverflows: a variant reported per application socket in ss -lnt as the Send-Q column. A persistent non-zero value there means that specific listener is undersized.
• TcpRetransSegs and TcpExtTCPLostRetransmit: retransmissions well above about 0.1 percent of segments suggest loss, not buffer starvation. Raising tcp_rmem will not fix loss.
• TcpExtPruneCalled and TcpExtRcvPruned: the receive buffer could not hold the window that was advertised. This is the counter that proves rmem_max is too low.
• netdev_max_backlog overruns: visible in /proc/net/softnet_stat as the second column per CPU. A rising count means the softirq path is dropping packets before they even reach a socket.
• nf_conntrack_count versus nf_conntrack_max: anything above roughly 70 percent utilisation is a countdown to "table full, dropping packet".

nstat -az | grep -Ei 'drop|retrans|prune|overflow|listen'
ss -lnt                                  # Send-Q column = accept queue high-water mark
awk '{print NR-1, $2, $3}' /proc/net/softnet_stat   # dropped / squeezed per CPU
cat /proc/net/netstat | grep -E 'TcpExt'

The discipline that matters is comparing counters before and after a change. A tuning change with no baseline is an opinion, not an experiment.

Queueing discipline, congestion control and NIC offloads

Socket buffers only help if the queueing below them does not add latency. The modern default on most distributions is fq_codel, which is a good general choice; fq is required by BBR and is the usual companion to it. For links where you also shape, cake handles bufferbloat better but is more expensive per packet.

# /etc/sysctl.d/11-tcp-congestion.conf
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_fastopen = 3          # client + server TFO
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_notsent_lowat = 16384

# confirm the kernel actually loaded the module
sysctl net.ipv4.tcp_available_congestion_control
modprobe tcp_bbr

On the receive side, do not forget the NIC itself. A 10G or 25G adapter with the default 512-entry receive ring will drop frames under burst load no matter what rmem_max says. Check and raise the ring, then steer interrupts across CPUs so a single core is not the bottleneck.

ethtool -g eno1                          # current vs maximum ring size
ethtool -G eno1 rx 4096 tx 4096          # raise to the maximum reported
ethtool -k eno1 | grep -Ei 'gro|gso|tso' # offloads, leave enabled
ethtool -L eno1 combined 8               # spread queues across cores
cat /proc/interrupts | grep eno1         # confirm IRQs land on different CPUs

# RPS/RFS for single-queue NICs
echo f > /sys/class/net/eno1/queues/rx-0/rps_cpus
sysctl -w net.core.rps_sock_flow_entries=32768

Memory ceilings: tcp_mem and the port space

Two less obvious parameters bite large servers. net.ipv4.tcp_mem is expressed in pages, not bytes, and governs the total memory the TCP stack may use across all sockets. If the third value is reached the kernel starts pruning sockets, which looks to applications like random connection resets. The ephemeral port range and the local reserved ports matter on any host that originates a lot of outbound connections, such as a proxy or a scrape worker.

# pages of 4 KiB: 786432 = ~3 GiB, 1048576 = ~4 GiB, 1572864 = ~6 GiB
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_max_tw_buckets = 2097152
net.ipv4.tcp_max_orphans = 262144
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.ip_local_reserved_ports = 5000-5100
sysctl -w net.ipv4.tcp_mem='786432 1048576 1572864'
cat /proc/net/sockstat              # sockets in use, TCP memory pages in use

Read /proc/net/sockstat under peak load: the TCP memory pages figure creeping towards the tcp_mem maximum is the warning that comes well before pruning starts.

A worked example: a reverse proxy dropping connections at peak

An nginx host fronting an API reports sporadic 502s during the daily peak. Nothing appears in the nginx error log, and CPU and memory look calm. The method is the same every time:

1. ss -lnt shows the listener with Send-Q at 511 while net.core.somaxconn is 128. The kernel is clamping the application's request.
2. nstat -az | grep ListenDrops confirms a rising drop count that tracks the 502 timestamps.
3. Raise net.core.somaxconn to 8192 and net.ipv4.tcp_max_syn_backlog to 8192, then also raise nginx's own listen ... backlog=8192 and the systemd unit's LimitNOFILE.
4. Re-measure with the same counters. If Send-Q now stays under a few hundred and the drop counter is flat, the change worked.

Note the order: kernel ceiling, application request, per-process file descriptor limit. Changing only one of the three moves the bottleneck rather than removing it.

Per-process limits that sysctl cannot reach

# systemd services: do not rely on /etc/security/limits.conf
systemctl edit nginx
#   [Service]
#   LimitNOFILE=1048576
#   LimitMEMLOCK=infinity

prlimit --nofile --pid $(pgrep -f 'nginx: worker' | head -1)
cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

For nginx specifically, worker_rlimit_nofile in nginx.conf must be at least as large as the sum of worker_connections across all workers, otherwise the effective limit is whichever value is smallest. Java services have the same trap with the file descriptor ceiling on the JVM process, and prlimit is the reliable way to see what a running process actually has.

Sizing buffers to the bandwidth-delay product, not to fashion

A socket buffer only has to hold one bandwidth-delay product (BDP) to keep a single connection busy. The BDP is the path bandwidth multiplied by the round-trip time, and that number - not a value copied from a 100G tuning guide - is what should decide rmem_max and wmem_max. A 1 Gbit link with a 20 ms RTT needs roughly 2.5 MB of window; the same link across a 200 ms satellite path needs about 25 MB. Copying a 64 MB maximum onto a 1G host wastes memory per connection and makes the OOM killer more likely, not less.

# BDP in bytes = bandwidth (bits/s) / 8 * RTT (seconds)
# 1 Gbit/s, 20 ms:   125000000  / 8 * 0.020 = 312500 bytes
# 10 Gbit/s, 5 ms:   1250000000 / 8 * 0.005 = 625000 bytes
# 10 Gbit/s, 100 ms: 1250000000 / 8 * 0.100 = 12500000 bytes

ping -c 20 10.0.0.20 | tail -1          # the RTT you are actually tuning for
ss -tin dst 10.0.0.20 | grep -E 'rtt|rwnd|cwnd|retrans'

ss -tin is the most useful command in this whole exercise: it prints the smoothed round-trip time, the congestion window and the receive window the kernel is using right now, so you can see whether the buffer you configured is the binding limit or whether the congestion window is. If cwnd sits far below rwnd, the bottleneck is loss and congestion control, and no sysctl will fix it. `retrans` in the same output tells you which of the two you are dealing with.

Re-measure, and then stop changing things

Change one group of keys, run the same measurement as the baseline, and compare the counters that justified the change. If those counters have not moved, revert the change. The most common failure mode in production tuning is not a bad value but a change that was never measured, and that therefore can never safely be reverted because nobody knows what it fixed. Set a review date, keep the drop-in files in version control beside the application configuration, and treat every number in them as a hypothesis the next measurement is allowed to disprove. Two or three focused changes that are understood beat twenty settings copied wholesale.

Making the change durable and reviewable

# one file per concern, numbered so the load order is deterministic
ls /etc/sysctl.d/
# 10-tcp-socket-buffers.conf  11-tcp-congestion.conf  20-conntrack.conf

sysctl --system                     # reload every drop-in in order
sysctl -n net.core.somaxconn        # read a single key back
systemd-sysctl --prefix=/etc/sysctl.d

Keep the drop-ins in configuration management alongside the application, and record the before/after sysctl -a dump with the change ticket. A tuning change shipped without a baseline and a rollback path is a future three-in-the-morning incident. Useful companions for the measurement half of the loop: the iperf3 testing guide, nginx stream module load balancing and Linux VXLAN VTEP configuration for overlay traffic whose MTU interacts with every buffer setting here.

原文链接：https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/tuning-the-network-performance_monitoring-and-managing-system-status-and-performance