Linux NIC Ring Buffer and Coalescing Tuning

Packet loss on a Linux host is not always a switch problem. A 25GbE link with an application that reports timeouts often has a driver-side ring buffer at the default 1024 descriptors and an interrupt coalescing setting tuned for a 1GbE era. The counters that prove it are visible in ethtool -S, and the fix takes three commands. This article walks through identifying ring buffer overflow, choosing coalescing values for throughput or latency, and making the settings survive a reboot.

Diagnose before changing anything

ip -s link show ens1f0
ethtool -S ens1f0 | grep -E "drop|miss|no_buffer|fifo" | grep -v ": 0"
ethtool -g ens1f0
ethtool -c ens1f0

rx_missed_errors and driver-specific rx_no_buffer_count mean the NIC had no free descriptor when a frame arrived and had to throw it away. That is a host-side drop, and no amount of switch tuning will help. Kernel-side drops show up separately as rx_dropped in ip -s link, or in /proc/net/softnet_stat where a rising second column means the softirq budget was exceeded and the backlog queue overflowed.

Raise the ring buffers

ethtool -g ens1f0          # note the "Pre-set maximums"
ethtool -G ens1f0 rx 8192 tx 8192
ethtool -g ens1f0          # confirm

The maximum is a hardware property, not a configuration choice; if the pre-set maximum is 1024, the adapter cannot help and the fix is fewer, bigger packets or a better NIC. Raising the ring costs kernel memory per queue, which on a 64-queue 100GbE adapter is real but almost never the limiting factor.

Coalescing: the throughput-versus-latency dial

Coalescing batches packets before raising an interrupt. More batching means fewer interrupts and higher throughput, at the cost of added latency for each packet waiting in the batch.

# Throughput oriented (bulk transfer, storage, training)
ethtool -C ens1f0 adaptive-rx on adaptive-tx on

# Explicit high-throughput values
ethtool -C ens1f0 rx-usecs 64 rx-frames 256 tx-usecs 64 tx-frames 256

# Low latency (trading, real-time, interactive)
ethtool -C ens1f0 adaptive-rx off adaptive-tx off
ethtool -C ens1f0 rx-usecs 0 rx-frames 1 tx-usecs 0 tx-frames 1

Adaptive mode is the right default for general workloads because the adapter self-tunes against observed traffic. Explicit values are for cases where you have measured the workload and know which side of the trade-off you want to be on.

Check the queues and IRQ affinity

ethtool -l ens1f0                 # current vs maximum channels
ethtool -L ens1f0 combined 16     # set queue count
grep -E "ens1f0" /proc/interrupts | awk '{print $1, $NF}'

With RSS, each receive queue has its own interrupt, and a single CPU handling many queues is the classic hidden bottleneck: throughput looks capped, mpstat -P ALL 1 shows one core at 100 percent softirq, and coalescing settings appear to do nothing. Spread the interrupts with irqbalance or explicit affinity, and keep a queue away from the NUMA node that is already busy with the application.

Make it persistent

nmcli connection modify ens1f0 ethtool.ring-rx 8192
nmcli connection modify ens1f0 ethtool.ring-tx 8192
nmcli connection modify ens1f0 ethtool.coalesce-adaptive-rx on
nmcli connection up ens1f0
ethtool -g ens1f0 && ethtool -c ens1f0

NetworkManager stores these in the connection profile, so they reappear after a reboot or a link bounce. Setting them by hand in a script works until the first unattended reboot, which is exactly when nobody is watching the counters.

The rest of the stack

• Backlog queue — net.core.netdev_max_backlog and net.core.netdev_budget control how much work softirq will do per pass; raise them before concluding the NIC is at fault.
• Socket buffers — net.core.rmem_max and net.ipv4.tcp_rmem cap what an application can ask for, and a large ring behind a small socket buffer just moves the drop point.
• Offloads — check with ethtool -k ens1f0. LRO is usually wrong for forwarding hosts, and disabling TSO/GRO is the standard step before taking a capture you intend to trust.

NIC tuning is one layer of a stack that also includes queueing on the switch. If drops occur on the wire rather than in the host, the congestion diagnosis path described in EOS queue counters and congestion troubleshooting and the shaping tools in Linux tc HTB traffic shaping are the next places to look.

原文链接：https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/network_troubleshooting_and_performance_tuning/tuning-network-adapter-settings