Jumbo Frames MTU Mismatch: Diagnosing Silent Drops

Jumbo frame problems present in a way that looks nothing like a network fault: the application works, ping works, but one host cannot copy a large file, or storage replication stalls only on the path through one particular switch. The reason is that MTU mismatches are directional and selective — large frames are dropped silently with no ICMP error when they are generated locally with the DF bit set. This article covers the diagnostic method that finds the offending device in minutes, the classic causes, and how to fix them without introducing fragmentation.

First, Understand the Failure Shape

The documented pattern looks like this: large pings with the DF bit set fail only to specific destinations, and the failure begins near the 1500-byte boundary. That is the signature of an endpoint or an intermediate device configured at 1500 while the rest of the path is at 9000.

The reason it is silent: when a frame arrives at a device whose MTU is smaller than the frame, the device drops it. It cannot fragment it if DF is set, and even when it can fragment, many platforms do not send an ICMP fragmentation needed message for locally originated traffic. So the source never learns.

Step 1 — Prove the MTU with DF-Bit Pings

Binary-search the payload size from one endpoint to the other. This is the fastest and most reliable test:

# Linux - 1472 payload = 1500 byte IP packet
ping -M do -s 1472 10.10.20.50
ping -M do -s 8972 10.10.20.50      # 9000 byte frame

# Windows
ping -f -l 1472 10.10.20.50
ping -f -l 8972 10.10.20.50

# Cisco IOS / IOS-XE
ping 10.10.20.50 size 8972 df-bit
ping 10.10.20.50 size 1472 repeat 5 df-bit

If 1472 succeeds and 8972 fails, something in the path is at 1500. If both fail including small sizes, it is not an MTU problem at all.

Step 2 — Walk the Path

Extend the ping hop by hop. From the originating host, ping the first L3 hop with DF at the jumbo size, then the next, and so on. The first hop that fails is behind the culprit. On switches, that means checking both directions: a device can forward a large frame in one direction and drop it in the other if the two paths traverse different hardware.

Classic Causes

• Endpoint NIC left at 1500. The most common case by a wide margin. A new server shipped with default MTU in a 9000 fabric.
• The management or out-of-band path. Simple Network Management and backup traffic often crosses a device still at 1500. If pings to the loopback succeed but to a data interface fail, check the intermediate device, not the endpoint.
• LAG member inconsistency. One member of a port-channel at 9000 and another at 1500 makes the fault intermittent: it fails only for flows hashed onto the wrong member.
• MTU on the wrong layer. On routers that separate L2 and L3 MTU, raising the interface MTU without raising the IP MTU on the subinterface has no effect on routed traffic.

Step 3 — Fix and Verify

! Cisco IOS-XE - system MTU affects all physical interfaces on many platforms
system mtu jumbo 9000
!
! Cisco IOS-XR / Nexus per-interface
interface TenGigabitEthernet 0/0/0
 mtu 9216

# Linux - make it persistent, not just runtime
ip link set dev ens5f0 mtu 9000
# /etc/network/interfaces:  mtu 9000
# nmcli: nmcli con mod "eth0" 802-3-ethernet.mtu 9000

Always re-run the DF-bit ping to both the near end and the far end after the change, and re-check every member of any port-channel on that path. A partially fixed MTU is worse than an obviously broken one because the fault becomes flow-dependent.

Sizing Rule

A 9000-byte MTU means 9000 bytes of IP payload including headers, so the largest ICMP payload you can send is 8972 (9000 minus 20 bytes IP and 8 bytes ICMP). If your switches are configured at 9216, you have 40 bytes of headroom which covers VLAN and MPLS label stacks and is the safer common choice in data centres.

Related reading on this site: MTU, MSS and PMTUD: Fixing Black-Hole Connections fundamentals, the protocol-specific analysis in IPv6 Path MTU Discovery and ICMPv6 Packet Too Big, and iSCSI Multipathing with multipathd: Tuning Guide where jumbo frames and path MTU interact directly.

原文链接：https://supportportal.juniper.net/s/article/Jumbo-frames-fail-to-specific-destinations-due-to-endpoint-MTU-mismatch