Linux VXLAN Configuration: ip link and Static VTEP Setup

VXLAN extends layer-2 segments over a layer-3 network by wrapping Ethernet frames in UDP (RFC 7348), and Linux has shipped a native VXLAN implementation in the kernel for years - no Open vSwitch required. With two ip link commands you can build a static VTEP pair that carries a whole L2 segment between servers or VMs across a routed fabric. This guide explains the Linux VXLAN device model, shows multicast and static-remote configurations with FDB management, covers bridging, MTU and offload, and explains how to persist the setup so it survives a reboot, based on the official kernel networking documentation.

The Linux VXLAN Device Model

A VXLAN device is a 1-to-N tunnel: unlike GRE it is not point-to-point. One interface can reach many remote VTEPs, and the kernel decides per frame where to send it. Three pieces of state drive that decision:

• The VNI - the 24-bit VXLAN Network Identifier in the header, the equivalent of a VLAN ID but with a large enough space to be globally unique. id 42 on the command line sets it.
• The underlay - the physical interface (dev eth1) and destination UDP port (normally 4789) used to reach the remote VTEPs.
• The forwarding table (FDB) - the mapping from inner MAC addresses to remote VTEP IPs, learned dynamically or populated statically.

The device learns remote endpoint IPs either dynamically like a learning bridge (receiving encapsulated traffic teaches it where a MAC lives) or from statically configured FDB entries. Traffic for unknown destinations is sent to a multicast group when one is configured, or dropped in pure unicast mode when there is no group and no static entry.

That last sentence is the most important operational detail: a VXLAN interface with no multicast group and no FDB entries will carry unicast traffic only for destinations it has already learned or been told about. Broadcast, unknown-unicast and multicast (BUM) traffic such as ARP requests has nowhere to go. This is exactly why small static deployments either use a multicast group on the underlay or populate the FDB by hand.

Basic Device Creation (Multicast Mode)

ip link add vxlan0 type vxlan id 42 group 239.1.1.1 dev eth1 dstport 4789
ip link set dev vxlan0 up

This creates vxlan0 with VNI 42, using multicast group 239.1.1.1 on eth1 for flood traffic and the IANA destination port 4789. The group parameter is what makes BUM traffic work: when the kernel needs to flood, it sends the frame unicast to the underlay address and lets multicast routing replicate it to every VTEP that joined the group. Older kernels defaulted to a different port, so always set dstport 4789 explicitly for interop with switches and other vendors.

Multicast mode needs the underlay network to route or switch the group between all participating hosts. On a flat L2 underlay that is usually free; on a routed fabric it requires PIM or an equivalent multicast capability, which is often the reason operators accept the extra bookkeeping of static FDB entries instead.

Static VTEP (Unicast) Configuration

Without multicast you must add a static FDB entry pointing the remote VNI at the peer VTEP IP, or use the remote keyword to set a default destination. The simplest point-to-point pair uses remote plus an address in the bridged segment:

# Host A (VTEP 10.0.0.1)
ip link add vxlan10 type vxlan id 100 remote 10.0.0.2 dstport 4789 dev eth0
ip addr add 192.168.200.1/24 dev vxlan10
ip link set vxlan10 up

# Host B (VTEP 10.0.0.2)
ip link add vxlan10 type vxlan id 100 remote 10.0.0.1 dstport 4789 dev eth0
ip addr add 192.168.200.2/24 dev vxlan10
ip link set vxlan10 up

Both hosts can now ping each other over the VXLAN, including ARP, because with a single remote the interface has a default destination for every frame, flooded or not. On Host A, adding remote 10.0.0.2 means "send everything you cannot deliver locally to this VTEP".

To carry VMs instead, slave vxlan0 to a Linux bridge (ip link set vxlan0 master br0) exactly like a VLAN trunk interface. The bridge then switches between the VM tap devices and the VXLAN device, and the VXLAN device handles encapsulation. With more than two hosts and no multicast, drop the remote keyword and populate the FDB explicitly for every remote VTEP, as shown below.

Bridging VXLAN into a Linux Bridge

The usual production layout puts the VXLAN device and the local VM or container interfaces into one bridge, so the overlay is invisible to the guests:

ip link add br-vx100 type bridge
ip link set vxlan10 master br-vx100
ip link set veth-a master br-vx100
ip link set br-vx100 up
bridge vlan show dev vxlan10

Two details decide whether this works. First, the bridge must be up before the VXLAN device can forward, and the VXLAN device's underlying interface dev eth0 must be up as well; if the underlay is down the tunnel comes up but black-holes traffic. Second, keep the VNI-to-bridge mapping one-to-one. Bridging two VXLAN devices with the same VNI into one bridge creates a loop through the underlay.

Managing the FDB

The forwarding database is where you tell the kernel which remote VTEP owns which MAC address. This is the manual replacement for the learning that multicast mode performs automatically:

# add / delete / show forwarding entries
bridge fdb add to 00:17:42:8a:b4:05 dst 192.19.0.2 dev vxlan0
bridge fdb delete 00:17:42:8a:b4:05 dev vxlan0
bridge fdb show dev vxlan0
ip -d link show vxlan0

bridge fdb show dev vxlan0 prints both the entries you added and any the kernel learned, with a self flag marking locally created entries. ip -d link show vxlan0 is the complementary view: it reveals the VNI, the destination port, the underlying device, and the learning and offload flags in one line, which makes it the first command to run when a tunnel is not passing traffic. Options such as nolearning, external and proxy are best reserved for integration with an external EVPN control plane; a static deployment should keep learning on so that replies populate the table on their own.

MTU, Underlay and Offload Considerations

Every VXLAN frame adds 50 bytes of encapsulation to the original frame: 8 bytes of outer UDP header, 8 of VXLAN header, 20 of outer IP, and 14 of outer Ethernet (plus any VLAN tags on the underlay). A guest with a 1500-byte MTU then produces 1550-byte frames on the wire. If the underlay or any device on the path only supports 1500, those frames are fragmented or dropped, and the symptom is the classic one: small pings and ARP work, SSH connects and then hangs on bulk data, and large transfers stall. Two standard fixes exist. Raise the underlay MTU to at least 1600 (jumbo frames end to end) and leave guests at 1500, or keep a 1500-byte underlay and reduce the guest MTU to 1450. Lowering the guest MTU is the safer choice on a shared network, because it depends on nothing but the guest itself.

By default the kernel hashes the UDP source port to spread entropy across ECMP paths - pinning is possible with the srcport parameter if a firewall or load balancer requires a fixed port, but avoid it on routed fabrics because it collapses the flow hashing that makes VXLAN scale. Check NIC offload support (ethtool --show-tunnels eth0); rx-udp_tunnel-port-offload improves performance by parsing the inner frames in hardware. Echo the setting back with ethtool -k eth0 | grep -i tunnel before trusting it.

Persisting the Configuration

Commands typed at a shell do not survive a reboot. On a server with ifupdown, write the VXLAN device into /etc/network/interfaces with a pre-up line:

auto vxlan10
iface vxlan10 inet manual
    pre-up ip link add vxlan10 type vxlan id 100 dstport 4789 dev eth0 || true
    post-down ip link del vxlan10 || true

The || true matters: without it, a second ifup on an already existing device fails and leaves the interface down. Systems running systemd-networkd use a .netdev file of Kind=vxlan with the same parameters, which removes the need for the pre-up trick entirely. Either way, verify after a reboot with ip -d link show rather than assuming the boot path succeeded.

Security and Edge Cases

VXLAN carries plain Ethernet inside UDP with no encryption or authentication. Anyone who can inject a packet to UDP port 4789 with the right VNI joins the segment. In a shared or untrusted underlay this is a real exposure, and the usual mitigations are an IPSec transport between VTEPs, a wireguard underlay, or restricting UDP 4789 with a host or fabric ACL to the known VTEP addresses. A static deployment where the peers are enumerated in the FDB is convenient here: because you already know the exact set of remote VTEP IPs, the ACL is simply that list.

A handful of other edges trip people up. A VXLAN interface inherits its MAC from a random address and can be pinned with address when an upstream switch needs a stable entry. Two VXLAN devices cannot share a VNI on the same host unless one is a duplicate for a different bridge, which is almost never intended. And overlays do not carry link-state: if the underlay route to a remote VTEP disappears, the tunnel stays administratively up and simply drops frames, so monitoring has to watch the underlay, not the tunnel's flag.

Verifying and Troubleshooting

ip -d link show vxlan10
bridge fdb show dev vxlan10
ip route get 192.168.200.2 dev vxlan10
tcpdump -i eth0 -nn udp port 4789 -c 10
ping -M do -s 1450 192.168.200.2

Read those five commands in order. ip -d link show proves the device exists with the right VNI, port and underlying interface. bridge fdb show proves the remote VTEP is known. ip route get proves the overlay route points at the VXLAN device rather than at the underlay. tcpdump on the physical interface proves that encapsulated packets are actually leaving with UDP port 4789, which separates a configuration problem from an underlay routing or firewall problem. The final ping -M do -s 1450 tests a 1478-byte frame with the don't-fragment bit set, which is the fastest way to confirm the MTU question on a new tunnel.

Related Guides on This Site

VXLAN overlays pair naturally with Linux bridging and VLANs: see VXLAN MTU and fragmentation: the 50-byte tax, VXLAN EVPN multi-site border gateway design, Linux VLAN tagging with ip link, and Docker networking drivers: bridge, macvlan and overlay.

原文链接：https://www.kernel.org/doc/html/latest/networking/vxlan.html