VXLAN MTU and Fragmentation: The 50-Byte Tax

VXLAN is a 50-byte tax on every packet, and one day that tax comes due. SSH and DNS work, the application health check passes, then a file upload or an image-heavy page hangs forever. Nothing is broken — the overlay's large packets are being dropped silently and Path MTU Discovery is not getting through. This guide explains the arithmetic, shows how to confirm the diagnosis in two commands, and lists the fixes in the order you should try them.

Where the 50 Bytes Go

Linux spells the same maths out in the VXLAN driver, where the headroom reservation is literally outer IP + UDP + VXLAN + inner Ethernet. Add an inner VLAN tag and the overhead grows by those 4 bytes; move to an IPv6 underlay and it becomes about 70 bytes.

The Arithmetic That Decides Your Day

Underlay MTU 1500:  VXLAN overlay MTU = 1500 - 50 = 1450
                    TCP MSS (IPv4)    = 1450 - 40 = 1410

Underlay MTU 9000:  VXLAN overlay MTU = 9000 - 50 = 8950
                    (jumbo underlay lets tenants keep MTU 1500 with headroom)

Other encapsulations for comparison on a 1500 underlay:
  IP-in-IP     20 bytes -> pod MTU 1480
  GRE          24 bytes -> MTU 1476
  VXLAN        50 bytes -> MTU 1450
  WireGuard    60 bytes -> MTU 1440

A 1500-byte frame from a guest becomes a 1550-byte frame on the wire, which is larger than the physical MTU. With DF set — which TCP sets — the underlay must either fragment it or drop it and send ICMP "fragmentation needed". If that ICMP is filtered (common with security policies that block all ICMP), you get a black hole: the sender never learns to reduce size.

The Two-Command Diagnosis

# 1. Find the real path MTU by bisecting with DF set
ping -M do -s 1472 10.244.3.17      # fails silently on a VXLAN overlay
ping -M do -s 1422 10.244.3.17      # succeeds  -> 1422 + 28 = 1450 path MTU
# largest working payload + 28 = path MTU of the overlay.

# 2. Check what MTU the interface claims
kubectl exec -it web-7d9f9 -- ip link show eth0
# mtu 1500  <- wrong inside a VXLAN overlay; should be 1450
ip link show eth0 ; ip link show vxlan10

The signature to recognise: small packets flow, large packets vanish. Ping is tiny and fits; a DNS query is tiny; the first full-size segment is where the failure begins. Anything that only breaks on downloads, uploads or video is an MTU problem until proven otherwise.

Fixes in the Right Order

1. Best: jumbo underlay. Set MTU 9000 (or 9216) on the physical NICs, switch ports and VTEP interfaces. The 50-byte tax then hides inside jumbo headroom, tenants keep a clean 1500-byte MTU, and there is room for stacked encapsulations such as VXLAN over IPsec.

   ip link set eth0 mtu 9000
   ip link set vxlan10 mtu 8950        # or leave the VTEP at 9000 in many designs

2. If the underlay must stay at 1500, lower the overlay interface MTU to physical - 50 (1450), or set it explicitly on the VTEP:

   ip link set vxlan10 mtu 1450

   Straightforward for VMs you control; in Kubernetes this is the CNI's MTU setting, so change it at the CNI (Calico, Cilium, Flannel all expose it) rather than inside individual pods.

3. Clamp TCP MSS so hosts negotiate a size that fits, even when PMTUD is broken:

   iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
            -j TCPMSS --clamp-mss-to-pmtu
   # or a fixed value: --set-mss 1410
   iptables -t mangle -L FORWARD -v -n | grep TCPMSS   # verify hits

4. Stop filtering the ICMP you depend on. Letting "fragmentation needed" through is cheaper than any of the workarounds above. If security policy forbids ICMP, you must clamp MSS — there is no third option.

Design Habit Worth Adopting

• Document the overlay MTU per fabric and per CNI; label it in the CMDB next to the underlay MTU.
• Test with a real large-payload check after every change window: a ping -M do -s 1400 from an edge host through the tunnel catches regressions before users do.
• Remember the multi-tenant case: keep tenant interfaces at a standard MTU and absorb the overlay overhead in the fabric, so nobody has to know about your encapsulation.
• Watch the counters: fragmented packets, ICMP unreachables generated and interface drops all move when the MTU is wrong.

相关阅读：TCP MSS 钳制与 PMTUD 排障、VXLAN EVPN 多站点 Border Gateway 设计 以及 Linux 网络命名空间与 veth 实践。

原文链接：How to Fix Fragmentation Issues with VXLAN Overlays