Geneve vs VXLAN: Overlay Encapsulation Compared

Every modern overlay — an EVPN data centre fabric, a Kubernetes CNI, a service mesh dataplane — puts one packet inside another. VXLAN has been the workhorse for a decade; Geneve arrived later with a flexible header designed for the same job but without VXLAN's hard-coded extension limits. Both are UDP-based, both are stateless in the network, and both add overhead that shows up in your MTU budget. This guide compares the two protocols at the byte level, looks at where each is actually deployed, and shows the commands that let you verify an overlay end to end rather than trusting that the encapsulation "just works".

Headers side by side

VXLAN encapsulates an inner Ethernet frame in UDP port 4789 with an 8-byte VXLAN header that carries a 24-bit VNI and a reserved field. That reservation has been reused for a few flags over the years, but the format is fixed. Geneve uses UDP port 6081 with a variable-length option header: the base header is 8 bytes, but it can carry TLV options for metadata such as a class-of-service hint, a security group identifier, or a vendor-specific tag.

VXLAN:  outer Eth | IP | UDP 4789 | VXLAN(8)    | inner Eth...   overhead 50 B (IPv4), 70 B (IPv6)
Geneve: outer Eth | IP | UDP 6081 | Geneve(8+n) | inner Eth...   overhead 50 B + option length

The practical consequence: Geneve can express intent that VXLAN cannot, without inventing a proprietary header. That is why cloud and container platforms chose it, and why hardware that inspects the overlay must be able to parse variable-length options.

MTU arithmetic you cannot skip

An overlay reduces the usable MTU by its overhead. With a 1500-byte physical path, a VXLAN tunnel over IPv4 leaves roughly 1450 bytes for the inner frame; over IPv6 underlay it leaves about 1430. Geneve with options can be smaller still.

# Linux VXLAN configuration with an explicit MTU
sudo ip link add vxlan100 type vxlan id 100 \
  local 10.0.0.1 remote 10.0.0.2 dstport 4789 dev eth0
sudo ip link set vxlan100 mtu 1450 up

# Geneve (as created by most CNIs)
ip -d link show genev_sys_6081 | grep -E "geneve|mtu"

# confirm end-to-end with DF-bit pings
ping -M do -s 1422 10.0.0.2        # succeeds if the path supports 1450
ping -M do -s 1472 10.0.0.2        # fails on a 1500-byte underlay

Raise the underlay to jumbo frames where the hardware allows, and clamp MSS on the tunnel interface for everything else. The failure mode when you forget is covered in the notes on TCP MSS clamping and PMTUD — small pings pass, bulk transfers stall.

Where each protocol is deployed

• VXLAN — EVPN-VXLAN data centre fabrics on Cisco, Arista, Juniper, Aruba and SONiC platforms, plus the classic leaf-spine fabric design. Hardware support is universal, and the 24-bit VNI space is ample.
• Geneve — Kubernetes CNIs (OVN-Kubernetes, Antrea), cloud provider overlays, and some SDN controllers that need to carry policy metadata with the packet.
• Both — multi-tenant environments where a CNI overlay runs over a fabric that already builds EVPN-VXLAN: you now have two encapsulations in series and must budget MTU for both.

# on a Linux host: see which overlays exist and how packets are wrapped
ip -d link show type vxlan
ip -d link show type geneve
bridge fdb show dev vxlan100 | head
tcpdump -ni eth0 udp port 4789 -c 5
tcpdump -ni eth0 udp port 6081 -c 5

Capturing a handful of packets on each port is the fastest way to answer "is the overlay actually encapsulating, or is traffic leaking as native frames?" — a question that becomes urgent the first time two tenants see each other's broadcast traffic.

Verification in a switch fabric

# Arista EOS
show vxlan vni
show vxlan address-table
show interfaces vxlan1 | include MTU|Vxlan

# Cisco NX-OS
show nve peers
show nve vni
show interface nve1 | include MTU

# Linux VTEP
ip -s link show vxlan100
bridge fdb show dev vxlan100 | grep -v permanent | wc -l

Check three things: the VNI-to-VLAN mapping exists on both ends, the remote VTEP address is reachable in the underlay, and the MAC learning entries are populated from the remote side rather than only locally. A VNI that is up with no remote FDB entries is a one-way overlay — usually a routing problem in the underlay, not an encapsulation problem.

Encapsulation compared: choosing between them

For a switch-based data centre, VXLAN with an EVPN control plane remains the pragmatic choice: no flooding, standards-based multihoming with ESI, and broad silicon support. For a container platform, let the CNI choose — most speak Geneve today and their control plane expects its options.

Operational checklist

• Document the encapsulation overhead for every overlay in the design and subtract it from the physical MTU before writing the underlay config.
• Verify that all devices in the path, including firewalls and load balancers, either permit the encapsulation ports or terminate the overlay — middleboxes that hash on the outer UDP header will break ECMP symmetry otherwise.
• Watch for VNI duplication when two teams build overlays independently; a shared VNI registry avoids the classic leak.
• Test failover with real traffic, not just pings, when overlay traffic crosses an EVPN multisite border gateway.
• For Linux VTEPs, keep the VXLAN interface MTU explicitly set rather than inherited — see the Linux VXLAN VTEP guide for the full configuration.

原文链接：https://www.rfc-editor.org/rfc/rfc8926.html