Jumbo Frame MTU Mismatch: Ping DF Bit Troubleshooting

Jumbo frames fail in the most deceptive way possible: small requests work, big transfers hang, and nothing in the logs says why. The classic symptom — connections establish but data stalls — is a Path MTU Discovery black hole, usually caused by an MTU mismatch somewhere along the path plus a firewall that drops the ICMP "fragmentation needed" messages. This guide shows how to isolate the mismatch with ping and the Don't Fragment bit, then fix it consistently across switches, SVIs and hosts.

Why MTU Mismatch Becomes a Black Hole

When a packet exceeds the next link's MTU, an IPv4 router can fragment it — unless the Don't Fragment (DF) bit is set. TCP sets DF to enable Path MTU Discovery: the router then drops the packet and sends back an ICMP type 3 code 4 message telling the sender to use smaller packets. If a firewall blocks that ICMP, the sender never learns, retransmits forever and the connection "works but transfers nothing". Tunnel overhead makes this worse — GRE subtracts 24 bytes, GRE over IPsec around 52, WireGuard 80, and PPPoE drops you to 1492.

The failure is mode-dependent, which is what makes it so confusing. A ping with default settings sends small packets and succeeds. An SSH login sends small packets and succeeds. Then the first bulk copy, backup or iSCSI read pushes a full MSS-sized segment, the DF bit is set, the oversize frame is dropped silently, and the transfer stalls at a crawl or hangs entirely. Users report "the network is broken" while monitoring shows green interfaces and near-zero error counters, because the drops happen on a device that is not counting them as errors at all.

There is also an IPv6 subtlety worth knowing. In IPv4, fragmentation is performed by routers unless DF is set. In IPv6, routers never fragment — only the source may, and the source learns the limit through ICMPv6 "Packet Too Big" messages. The mechanism is effectively always-on PMTUD. That means an IPv6 path with a blocked ICMPv6 type 2 message fails immediately and completely, whereas the IPv4 equivalent might limp along via fragmentation on some hops.

Finding the Path MTU with Ping and DF

Measure the real end-to-end MTU with DF set and the ICMP payload sized correctly (add 28 bytes for IP+ICMP headers):

# Linux: 1472 + 28 = 1500 (standard MTU)
ping -c 3 -M do -s 1472 8.8.8.8
# 8972 + 28 = 9000 (jumbo)
ping -c 3 -M do -s 8972 STORAGE_SERVER_IP
# Windows: ping -f -l 8972 <target>

If 8972 fails but 1472 succeeds, the path is not consistently jumbo-capable. Bisect between the two sizes to find the exact ceiling — for example 1450 succeeding and 1500 failing pinpoints the MTU. Crucially, always test with DF: without it the router fragments happily and the ping succeeds even when the real path MTU is smaller.

A quick bisection loop saves a lot of guesswork. Start at 1472 (1500 MTU) and step up: 1472, 4472 (4500 MTU), 8972 (9000 MTU), 9172 (9200 MTU), 9192 (9216 MTU). The largest value that succeeds plus 28 is your effective path MTU. On Linux you can do it in one line:

for s in 1472 4472 8972 9172 9192; do
  ping -c 1 -W 2 -M do -s $s 10.10.20.5 >/dev/null 2>&1 \
    && echo "$s OK (MTU $((s+28)))" || echo "$s FAIL"
done

Two practical details about the probe itself. First, some devices rate-limit or filter ICMP entirely, so a failing DF ping is evidence of a possible MTU problem but not proof — combine it with the DF-bit-set tracepath or traceroute --mtu output, which walks hop by hop and reports where the size is refused. Second, always test in both directions and from both endpoints. A successful DF ping from server A to server B says nothing about B to A.

Where the Mismatch Usually Lives

• Switch port MTU — the port is still at 1500 while the host is at 9000.
• SVI MTU — Layer 3 VLAN interfaces need their own ip mtu; the L2 port MTU alone is not enough for routed traffic.
• System MTU — on Cisco Catalyst the global system mtu change only applies after a reload.
• Tunnel/VPN interfaces — encapsulation overhead was never subtracted from the MTU.
• VLAN subinterfaces on Linux — a VLAN interface inheriting the parent MTU while the parent was changed.
• Virtual switch and hypervisor uplinks — a vSwitch or bridge defaults to 1500 even when the physical NIC was raised to 9000.
• Overlay networks — VXLAN adds 50 bytes and Geneve can add more, so an underlay at exactly 9000 will still drop inner jumbo frames.

On Cisco IOS-XE, check the hardware side with show controllers ethernet-controller Gi1/0/1 | include ValidOverSize — rising oversize counters prove frames bigger than the port MTU are arriving. Cisco Catalyst 9300-specific MTU troubleshooting is covered in depth by the network-switch.com engineering article.

On Nexus and Arista the equivalent counters are in the interface error statistics. Watch for FramesTooLong or giant counters, and be aware that on some ASICs these counters are cleared by a show command that resets them — take a reading, wait, and take another rather than reading once.

Layer 3 Interfaces and the Routed Path

The most commonly missed case is traffic that is switched correctly on Layer 2 but routed across an SVI. The physical port MTU governs frame size on the wire; the SVI's ip mtu governs fragmentation and PMTUD on routed packets. If the port is 9000 and the SVI is 1500, hosts on the same VLAN can talk at jumbo size but any traffic crossing the SVI is limited to 1500 — and because ICMP is often filtered, this shows up as a black hole for cross-VLAN traffic only.

! Verify both the port MTU and the routed MTU
Switch# show interface GigabitEthernet1/0/1 | include MTU
Switch# show interface Vlan10 | include MTU
Switch# show ip interface Vlan10 | include MTU

! Align them
Switch(config)# interface Vlan10
Switch(config-if)# mtu 9000
Switch(config-if)# ip mtu 9000

Note that mtu and ip mtu are separate settings on IOS. The mtu command sets the maximum frame size including the Layer 2 header; ip mtu sets the maximum IP packet size that the interface will forward without fragmenting. For the common case, keep them synchronized with the 14-byte Ethernet header offset in mind if a platform reports them differently.

Fixing It: The All-or-Nothing Rule

Jumbo frames require every device in the path to support them, so either standardize 9000 (or 9216) end-to-end across hosts, switches and SVIs, or keep 1500 and fix the tunnel endpoints instead:

! Cisco IOS - consistent jumbo deployment
Switch(config)# system mtu 9000
Switch(config)# interface Vlan10
Switch(config-if)# ip mtu 9000
Switch(config)# interface GigabitEthernet1/0/1
Switch(config-if)# mtu 9000
! Linux host
sudo ip link set ens3 mtu 9000
! TCP MSS clamp for tunneled/broken paths
R1(config-if)# ip tcp adjust-mss 1360

After the change, re-run the DF ping in both directions — asymmetric success (A to B works, B to A fails) is the signature of a one-sided mismatch. For storage networks that depend on jumbo frames, also verify the NIC and switchport of every hop; our ethtool diagnostics guide covers the host side and the SFP checklist covers the physical layer beneath it.

MSS Clamping: The Practical Workaround

When you cannot raise the MTU on every device — a managed circuit, a customer firewall, or a third-party WAN you do not control — clamp the TCP maximum segment size at the boundary:

! Clarify fragments that must traverse a 1500 path
R1(config-if)# ip tcp adjust-mss 1460
! Tunnel endpoint: subtract GRE + IPsec overhead
R1(config-if)# ip tcp adjust-mss 1360
! Junos equivalent
user@R1# set interfaces ge-0/0/0 unit 0 family inet tcp-mss 1360

Clamping works because the TCP endpoints see a smaller MSS and simply never generate an oversized segment, so PMTUD never has to be consulted and a blocked ICMP path becomes harmless. ip tcp adjust-mss rewrites the MSS option in passing SYN packets in both directions. It only helps TCP — UDP-based protocols such as iSCSI with a manually configured jumbo MTU, or NFS over UDP, get no benefit and need the MTU corrected properly. The wider subject of PMTUD, MSS and black-hole diagnosis is covered in our MTU, MSS and PMTUD troubleshooting guide.

Overlays Make It Worse

If the failing path runs over VXLAN, remember the encapsulation tax. VXLAN adds roughly 50 bytes (outer Ethernet 14, IP 20, UDP 8, VXLAN 8), so an inner frame of 9000 needs a 9050-byte outer frame on the underlay. A fabric built at exactly 9000 will fragment or drop every jumbo-sized encapsulated packet. Size the underlay generously — many operators run 9216 on the physical fabric — and verify the inner path MTU independently of the outer. Our VXLAN MTU and fragmentation article covers the arithmetic in detail.

The same logic applies to WireGuard, where the header is 60 bytes plus the outer IP/UDP, and to IPsec tunnels where the overhead depends on the cipher and mode. For WireGuard specifically, the default 1420-byte MTU is a safe assumption but not a guaranteed one, and handshake failures on large packets are a classic symptom. See our WireGuard MTU troubleshooting guide.

A Repeatable Diagnostic Sequence

1. Confirm the symptom shape: small traffic fine, bulk traffic stalls. That alone points at MTU rather than a link error.
2. Run a DF-set ping at 8972 from source to destination. If it fails, you have your answer; if it succeeds, the problem is not a simple MTU mismatch and you should look at offload or TCP window settings.
3. Bisect to the exact payload size that fails and convert to an MTU.
4. Walk the path with traceroute --mtu and find the hop where the size is refused.
5. Inspect that device: port MTU, SVI MTU, tunnel MTU, overlay overhead.
6. Fix the smallest consistent value across all devices, or clamp MSS if you cannot control the far end.
7. Re-test bidirectionally, then verify with a real workload — a 1 GB file copy or an iSCSI read — not just a ping.

For storage and NVMe-oF paths where jumbo frames are a hard requirement rather than an optimisation, the endpoint configuration matters as much as the fabric; our NVMe over TCP setup guide covers the host-side MTU and queue configuration that has to match.

Common Pitfalls

• Changing the MTU on the switch but not rebooting where system mtu requires a reload to take effect.
• Testing without the DF bit, which makes every path look healthy.
• Setting the host NIC to 9000 but leaving the virtual switch at 1500 in a virtualised environment.
• Adding jumbo frames on the storage network but forgetting the management or backup path that crosses the same SVI.
• Assuming both directions are symmetric. Always verify the reverse path.
• Relying on a single ping probe. Repeat it — an intermittent path may pass once.

原文链接：https://packetmentor.com/topics/mtu-fragmentation