VXLAN MTU and Fragmentation: The 50-Byte Tax

VXLAN is a 50-byte tax on every packet, and one day that tax comes due. SSH and DNS work, the application health check passes, then a file upload or an image-heavy page hangs forever. Nothing is broken — the overlay's large packets are being dropped silently and Path MTU Discovery is not getting through. This guide explains the arithmetic, shows how to confirm the diagnosis in two commands, and lists the fixes in the order you should try them.

Where the 50 Bytes Go

Linux spells the same maths out in the VXLAN driver, where the headroom reservation is literally outer IP + UDP + VXLAN + inner Ethernet. Add an inner VLAN tag and the overhead grows by those 4 bytes; move to an IPv6 underlay and it becomes about 70 bytes.

The Arithmetic That Decides Your Day

Underlay MTU 1500:  VXLAN overlay MTU = 1500 - 50 = 1450
                    TCP MSS (IPv4)    = 1450 - 40 = 1410

Underlay MTU 9000:  VXLAN overlay MTU = 9000 - 50 = 8950
                    (jumbo underlay lets tenants keep MTU 1500 with headroom)

Other encapsulations for comparison on a 1500 underlay:
  IP-in-IP     20 bytes -> pod MTU 1480
  GRE          24 bytes -> MTU 1476
  VXLAN        50 bytes -> MTU 1450
  WireGuard    60 bytes -> MTU 1440

A 1500-byte frame from a guest becomes a 1550-byte frame on the wire, which is larger than the physical MTU. With DF set — which TCP sets — the underlay must either fragment it or drop it and send ICMP "fragmentation needed". If that ICMP is filtered (common with security policies that block all ICMP), you get a black hole: the sender never learns to reduce size.

The Two-Command Diagnosis

# 1. Find the real path MTU by bisecting with DF set
ping -M do -s 1472 10.244.3.17      # fails silently on a VXLAN overlay
ping -M do -s 1422 10.244.3.17      # succeeds  -> 1422 + 28 = 1450 path MTU
# largest working payload + 28 = path MTU of the overlay.

# 2. Check what MTU the interface claims
kubectl exec -it web-7d9f9 -- ip link show eth0
# mtu 1500  <- wrong inside a VXLAN overlay; should be 1450
ip link show eth0 ; ip link show vxlan10

The signature to recognise: small packets flow, large packets vanish. Ping is tiny and fits; a DNS query is tiny; the first full-size segment is where the failure begins. Anything that only breaks on downloads, uploads or video is an MTU problem until proven otherwise.

Fixes in the Right Order

1. Best: jumbo underlay. Set MTU 9000 (or 9216) on the physical NICs, switch ports and VTEP interfaces. The 50-byte tax then hides inside jumbo headroom, tenants keep a clean 1500-byte MTU, and there is room for stacked encapsulations such as VXLAN over IPsec.

   ip link set eth0 mtu 9000
   ip link set vxlan10 mtu 8950        # or leave the VTEP at 9000 in many designs

2. If the underlay must stay at 1500, lower the overlay interface MTU to physical - 50 (1450), or set it explicitly on the VTEP:

   ip link set vxlan10 mtu 1450

   Straightforward for VMs you control; in Kubernetes this is the CNI's MTU setting, so change it at the CNI (Calico, Cilium, Flannel all expose it) rather than inside individual pods.

3. Clamp TCP MSS so hosts negotiate a size that fits, even when PMTUD is broken:

   iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
            -j TCPMSS --clamp-mss-to-pmtu
   # or a fixed value: --set-mss 1410
   iptables -t mangle -L FORWARD -v -n | grep TCPMSS   # verify hits

4. Stop filtering the ICMP you depend on. Letting "fragmentation needed" through is cheaper than any of the workarounds above. If security policy forbids ICMP, you must clamp MSS — there is no third option.

Design Habit Worth Adopting

• Document the overlay MTU per fabric and per CNI; label it in the CMDB next to the underlay MTU.
• Test with a real large-payload check after every change window: a ping -M do -s 1400 from an edge host through the tunnel catches regressions before users do.
• Remember the multi-tenant case: keep tenant interfaces at a standard MTU and absorb the overlay overhead in the fabric, so nobody has to know about your encapsulation.
• Watch the counters: fragmented packets, ICMP unreachables generated and interface drops all move when the MTU is wrong.

Why Jumbo Frames Do Not Always Save You

Jumbo framing is a per-hop property, and a single 1500-MTU hop in the middle of the underlay breaks it for everyone behind it. Before pushing MTU 9000 across a fabric, confirm the entire path: access switch, spine, WAN handoff, firewall, load balancer and any hypervisor bridge or virtual switch. A common trap is a firewall or load balancer that quietly rewrites MTU back to 1500, or a cloud virtual NIC that caps out far lower than the physical link. The symptoms are the same black hole you were trying to avoid, only now they show up intermittently and only for large flows.

Cloud and Tunnel Underlays Break the Assumption

Compute the budget from the inside out. VXLAN riding inside a 1500-byte WireGuard tunnel is 1500 - 60 - 50 = 1390 for the guest, so the guest MTU should be 1390, not 1450. Document that arithmetic on the same page as the CNI or VTEP configuration so the next engineer does not have to rediscover it.

Kernel and Offload Behaviour on Linux

The VXLAN driver reserves headroom explicitly — outer Ethernet, IP, UDP and the VXLAN header — so the kernel knows exactly how much room the encapsulation needs. Two knobs matter in practice:

• needed_headroom on the VTEP link is what lets the stack build the outer headers without reallocating the skb. If a lower-MTU device cannot supply that headroom, packets are dropped rather than fragmented.
• TSO/GSO/GRO offload on the physical NIC lets the kernel hand large super-packets to the hardware, which segments them to the physical MTU after encapsulation. This is why a mis-set MTU can look healthy on a fast host with offloads enabled and fail the moment the flow reaches a device that does not preserve that segmentation.

# what the VTEP interface believes, and what the kernel reserved
ip -d link show vxlan10 | grep -E 'mtu|vxlan'
ethtool -k eth0 | grep -E 'tso|gso|gro|tx-udp-segmentation'

# temporarily disable offloads to see whether the host path reproduces the failure
ethtool -K eth0 tso off gso off gro off

Turning offloads off is a diagnostic step rather than a fix, but it is one of the fastest ways to prove that MTU, and not hardware acceleration, is the culprit.

Reading the Counters That Confirm It

The failure leaves a fingerprint in the interfaces and on the ICMP path.

ip -s link show eth0          # look for TX drops and oversized/reassembly errors
ip -s link show vxlan10
netstat -s | grep -i -E 'frag|reassembl|too big'

# on a router that sits in the path
show interfaces counters errors
show ip icmp counters          # ICMP unreachable and fragmentation-needed generated

If the underlay router is generating "fragmentation needed" but the sender keeps sending full-size segments, something is filtering that ICMP between the two. Capture to prove it:

tcpdump -ni eth0 'icmp and icmp[icmptype] == icmp-unreach and icmp[icmpcode] == 4'
tcpdump -ni eth0 'udp port 4789 and greater 1472'

MSS Clamping in Depth

Clamping rewrites the MSS option in TCP SYN packets as they transit the router, so both endpoints negotiate a segment size that already accounts for the encapsulation. It works even when PMTUD is broken, which is why it is the most reliable workaround. The practical rules are these:

# clamp to the discovered path MTU (interface MTU minus headers)
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN          -j TCPMSS --clamp-mss-to-pmtu

# or pin a value that matches a 1450 overlay: 1450 - 40 = 1410 for IPv4
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN          -j TCPMSS --set-mss 1410

# IPv6 segments carry a larger header pair
ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN           -j TCPMSS --set-mss 1390

# verify the rule is actually matching
iptables -t mangle -L FORWARD -v -n | grep -E 'TCPMSS|pkts'

Two caveats. First, clamping only helps TCP; UDP-based protocols such as some VPNs and DNS-over-UDP large responses still depend on a correct MTU or on the application's own probing. Second, clamping on the wrong interface silently does nothing, because the rule has to see the SYN in the direction of the large flow.

MTU in Kubernetes: Cilium and Calico

Most Kubernetes MTU pain comes from a CNI installed with the default 1500 while the nodes sit on a VXLAN or IPsec overlay. Encapsulation mode differs by CNI; the MTU handling is what breaks in production:

# Calico: read and set the tunnel MTU explicitly
calicoctl get felixconfiguration default -o yaml
# mtu: 1450      (or 0 to auto-detect from the fabric)

# Cilium: check auto-detection, or pin the value
cilium config view | grep -i mtu
helm upgrade cilium cilium/cilium --set MTU=1450

# what a pod actually sees
kubectl exec -it web-7d9f9 -- ip link show eth0

Cilium can auto-detect the MTU from the default-route interface, which usually yields the right number for VXLAN; Calico can be told to auto-detect as well, but many installs are left at 1500. Whichever you use, after any change re-run the payload test pod-to-pod across nodes — node-to-node success is not proof that pod-to-pod works.

Verification Runbook

1. From a source host inside the overlay, run ping -M do -s 1422 <dest> (1422 + 28 = 1450) and bisect upward until it fails.
2. Confirm the interface MTU on both endpoints matches the discovered path MTU, not 1500.
3. Check the underlay routers for packets dropped due to MTU and for ICMP type 3 code 4 being generated.
4. Confirm the sender actually receives those ICMP messages; if not, find the ACL that filters them.
5. Add MSS clamping if ICMP cannot be unfiltered, then re-test the original large transfer.
6. Record the numbers — underlay MTU, overlay MTU, MSS — next to the tunnel definition so nobody has to rediscover them.

Frequently Asked Questions

Can I just raise the MTU to 9000 everywhere? Only if every device on every possible path supports it, including cloud handoffs and security appliances. A single 1500-MTU hop reintroduces the black hole.

Why does ping work while the file transfer still hangs? Ping size is not payload size. A 1400-byte ping only proves that 1400-byte packets survive; the failure may begin at 1451. Bisect with the DF bit set.

Is fragmentation ever the right answer? Rarely. IPv4 routers may fragment, but IPv6 routers do not, and fragmentation taxes the receiver with reassembly. Lowering the MTU or clamping MSS is almost always better.

Does this apply to WireGuard and IPsec as well? Yes — every encapsulation has overhead. The WireGuard guide linked above covers the 60-byte case; IPsec has its own variable overhead depending on the cipher and mode.

相关阅读：TCP MSS 钳制与 PMTUD 排障、VXLAN EVPN 多站点 Border Gateway 设计 以及 Linux 网络命名空间与 veth 实践。

原文链接：How to Fix Fragmentation Issues with VXLAN Overlays