TCP Tuning for High Bandwidth-Delay Product Links

When a 10 Gbps link between two data centres delivers 2 Gbps on a single TCP stream, the bottleneck is usually not the link, the switch or the application — it is the receive window. TCP flow control limits the sender to the amount of data the receiver has advertised it can buffer, and on a long path that limit bites well before the link capacity does. The fix is arithmetic, not guesswork: calculate the bandwidth-delay product, set the socket buffers to cover it, and confirm that window scaling is doing its job. This article walks through the measurement, the settings, and the verification.

Calculate the bandwidth-delay product first

The BDP is the amount of data that must be in flight to keep the pipe full:

BDP (bytes) = connection speed (bytes/s) x latency (seconds)

# 10 Gbps with 17 ms round-trip latency
# 10e9 / 8 = 1.25e9 bytes/s
# 1.25e9 x 0.017 s = 21,250,000 bytes approx 21 MB in flight

Measure the latency with a real test, not with a datasheet: ping a representative host several times and use the average round-trip time. A window smaller than the BDP caps throughput no matter how fat the link is, which is why the same server performs perfectly on a campus LAN and poorly to a remote site.

The window scaling problem

TCP originally supported a maximum window of 64 KiB. To exceed it, the window scaling option shifts the advertised value left by up to 14 bits during the three-way handshake, allowing windows in the gigabytes. Scaling is negotiated at connection setup by both ends, and if either side disables it the connection is capped at 64 KiB for its entire lifetime. The scale factor is fixed at handshake time, so an application cannot widen it later.

Red Hat's own measurements illustrate the stakes: on a 1 Gbps link with 1.5 ms round-trip time, throughput is roughly 630 Mbps with window scaling enabled and about 380 Mbps with it disabled. On higher-latency paths the gap becomes far larger.

# check what the kernel is willing to negotiate
sysctl net.ipv4.tcp_window_scaling
sysctl net.ipv4.tcp_rmem
sysctl net.ipv4.tcp_wmem

# observe the negotiated window on a live connection
ss -tin dst 192.0.2.20

ss -tin prints the current send and receive window limits for a connection, which is the fastest way to see whether the kernel is limiting you or the peer is.

Setting system-wide buffers

# /etc/sysctl.d/10-tcp-socket-buffers.conf
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_congestion_control = bbr

sysctl -p /etc/sysctl.d/10-tcp-socket-buffers.conf

The three values in tcp_rmem and tcp_wmem are minimum, default and maximum, and autotuning moves the actual size between default and maximum as conditions allow. Setting only the maximum is usually right: it lets TCP grow the window when the path supports it without pre-allocating memory for every socket. Note the dependency: raising the maximum above 64 KiB is pointless if window scaling is off.

Verify with iperf3 and a controlled test

# server side
firewall-cmd --add-port=5201/tcp
iperf3 --server

# client side -- single stream, 60 seconds
iperf3 --time 60 --client 192.0.2.1
# [  5] 0.00-60.04 sec  101 GBytes   14.4 Gbits/sec   receiver

A single-stream test measures the path as one connection sees it, which is the number that exposes window limits. Then run a parallel test with --parallel 8: if aggregate throughput is far higher than the single stream, the path is fine and the per-connection window is the constraint. That comparison is the single most useful diagnostic in this whole area.

Test with the parameters that match the real workload — same MTU, same congestion control, same number of connections — and remember that a stream test through the application may be limited by the application itself. Isolate the network first with iperf3, then look at the application.

Where else packets die

TCP tuning will not fix a receive path that is dropping packets in the kernel or the NIC. If throughput improves but stays below expectation, check ring buffers and coalescing before going further — the commands are in ethtool ring buffer and coalescing tuning. In environments with several tenants on one link, shaping or traffic classes above the transport layer may also be limiting throughput; see Linux tc HTB traffic shaping. And when the complaint is about a path rather than a host, measure the network's own view of latency and reachability with the probes in Cisco IP SLA and object tracking before changing host parameters — the arithmetic only helps if the path really does have the latency you measured.

Finally, tune both ends. A perfectly tuned sender talking to a receiver with a 64 KiB window and window scaling disabled will still run at LAN speeds on a WAN path, and the counters on the sender will show it waiting rather than retransmitting.

原文链接：https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/network_troubleshooting_and_performance_tuning/tuning-tcp-connections-for-high-throughput