Cisco EVPN/VxLAN Multisite: Troubleshooting Commands

2026年8月26日 LOG admin

EVPN/VxLAN 跨数据中心（Multisite）场景下，主机无法互访是最常见的故障。本文基于 Cisco TAC 官方文档，以 DC-1 主机 172.20.2.10 访问 DC-2 主机 172.20.2.100 为例，从控制平面（MAC/IP 路由、BGP EVPN 通告、ARP 表）到数据平面（抓包定位）给出完整的逐跳排查命令集，适用于 Nexus 9000 系列。

Why Multisite EVPN/VXLAN Breaks Where a Single Fabric Works

A single EVPN-VXLAN fabric is a closed system: every leaf shares one underlay, one BGP EVPN session mesh, and one set of route targets. A multisite deployment breaks that symmetry on purpose - each site has its own fabric, its own underlay, and its own autonomous system, and the only thing tying them together is a pair of border gateways. Every failure mode in multisite troubleshooting traces back to one of four seams: the BGP EVPN session between sites, the next-hop preservation rule for type-2 and type-5 routes, the L3 VNI and route-target mapping, or the data path across the border gateway.

The practical consequence is that you cannot debug a multisite problem from inside one site. Host 172.20.2.10 in DC-1 that cannot reach 172.20.2.100 in DC-2 may have a perfectly healthy local fabric: MAC learned, route advertised to the spine, ARP resolved. The break is in the seam. The commands below walk outward from the source host, through the local leaf, across the border gateway, and back into the destination site, proving one seam at a time.

The Reference Topology

Two sites, DC-1 and DC-2, each with a leaf-spine fabric and a pair of border gateways (BGWs). Within each site the IGP is OSPF or IS-IS and BGP EVPN is the address-family carrying MAC and IP reachability. Between sites the BGWs form an eBGP EVPN session over the underlay, or over a dedicated inter-site link. Tenant subnets are stretched with a Layer 2 VNI for ARP and MAC, and routed with a Layer 3 VNI (a symmetric IRB design). The two hosts under test are 172.20.2.10 in DC-1 and 172.20.2.100 in DC-2, both in the same stretched subnet but in different physical sites.

That last fact is what makes this hard: the hosts share a subnet and therefore share a broadcast domain logically, but their traffic must cross a routed, encapsulated transit. Any mismatch in route targets, VNI numbering or next-hop handling produces a partial failure - ARP resolves, pings sometimes work, and everything looks green in show ip route.

控制平面验证

1. 源节点基础检查

ToLeaf1# show ip interface brief | exclude down
ToLeaf1# ping 172.20.2.1

2. 首跳 leaf 的 MAC/ARP 学习

leaf1# show mac address-table
leaf1# show ip arp vrf all

确认源主机的 MAC（如 0035.1ac1.37c2）与 IP 已通过 Vlan1000 的 Eth1/23 学习到。

3. MAC 路由注入 BGP EVPN

leaf1# show bgp l2vpn evpn 0035.1ac1.37c2

关注 [2] 类路由（MAC/IP 通告）的下一跳、RT 与通告对象——MAC-only 路由与 MAC-IP 路由应分别通告给 spine 对等体。

4. leaf 到 spine 的通告核对

leaf1# show bgp l2vpn evpn neighbors 192.168.1.1 advertised-routes

Control Plane Cheat Sheet: What Each Show Command Proves

It helps to know exactly what question each command answers, because in multisite the same output can mean two different things depending on where you run it.

show mac address-table proves local learning. If the source MAC is missing here, nothing downstream can work - the fault is host, cable, port VLAN or access port.
show ip arp vrf all proves Layer 3 resolution inside the tenant VRF. A MAC without an ARP entry means ARP never completed, usually because the gateway or the remote host is not answering.
show bgp l2vpn evpn <mac> proves the route exists in the local BGP table and shows its type, next hop and extended communities. Compare the route target against the import list of the destination VNI.
show bgp l2vpn evpn neighbors <peer> advertised-routes proves the route was offered to the peer. Empty advertised-routes against a healthy table is a policy problem - a route-map, an outbound filter or a missing send-community extended.

Run the same four checks at the border gateway. A route present at the leaf but absent at the BGW means the in-site BGP EVPN session filtered or never received it; a route present at the BGW but absent at the far site's BGW means the inter-site session, the route target rewrite or the next-hop policy is at fault.

数据平面排查

Monitor Session 抓包

leaf1(config)# monitor session 1 type erspan-source
leaf1(config-erspan-src)# source interface ethernet 1/23
leaf1(config-erspan-src)# destination ip 10.1.1.1

在源 leaf、目的 leaf 与边界网关（BGW）分别抓包，用 Wireshark 核对 VXLAN 头、内层以太网头与 ICMP 负载，判断封装/解封装发生在哪一跳。

关键检查点

BGW 是否把 EVPN 路由从本 site 重通告到对端 site（核对 next-hop 是否保留）；
目的 leaf 的 show mac address-table 是否出现经 nve1 学习到的远端 MAC；
两端 VNI、RT 是否一致，L3 VNI 是否映射正确。

Reading a VXLAN Capture: The Headers That Matter

A captured VXLAN frame is five headers deep, and multisite faults usually live in the third one. From the outside in:

Outer Ethernet - source and destination are the tunnel endpoints (the VTEP loopbacks or the inter-site interfaces), not the hosts. If this never appears, no encapsulation is happening.
Outer IP and UDP - destination port 4789 identifies the VNI to the receiving VTEP. Check that the outer source IP matches the loopback the receiving switch expects as a next hop; an asymmetric pair of outer source IPs is a classic cause of one-way traffic.
VXLAN header - the 24-bit VNI field. A wrong VNI means the frame is decapsulated into the wrong bridge domain or VRF and the payload is dropped silently.
Inner Ethernet - the real host MACs. Seeing the inner frame arrive at the far leaf but no reply proves the problem is on the return path or in the destination's own forwarding.
Payload - ICMP echo request or reply. Matching opaque bytes between request and reply lets you prove which frames pair up across the tunnel.

Capture at three points - the source leaf's access port, the BGW's inter-site interface, and the destination leaf's access port - and compare. Frames present at point one and two but absent at point three localise the failure to the far fabric. Frames present at two but with a rewritten outer source localise it to next-hop policy.

Top Root Causes and Their Fixes

Next hop not preserved across the border. The BGW must advertise the remote leaf's VTEP as next hop so the receiving site can build the tunnel directly, or it must be the next hop itself while also withdrawing its own next-hop self rule consistently. Mixed behaviour between the two BGWs of a pair produces intermittent connectivity that clears and returns with each BGP update.
Route target mismatch. The type-2 routes must carry the RT that the remote L2 VNI imports, and type-5 routes must carry the L3 VNI RT. A single missing RT on the BGW's egress policy breaks one direction only.
VNI mismatch. Site-local VNIs do not have to match everywhere - that is the point of a translation layer - but the mapping must exist on both BGWs and be symmetric.
Underlay MTU. A VXLAN header adds 50 bytes to each frame. If the inter-site link runs a smaller MTU than the host-facing ports, large frames (or large ARP replies with many endpoints) vanish while pings succeed. See the dedicated note on this below.
Asymmetric ARP. Two hosts that can ping each other but cannot pass TCP often have an ARP entry learned from one path and used on another.

Step-by-Step Verification Checklist

leaf1# show nve vni
leaf1# show nve peers
leaf1# show bgp l2vpn evpn summary
BGW1# show bgp l2vpn evpn summary
BGW1# show bgp l2vpn evpn route-type 2
BGW1# show bgp l2vpn evpn 172.20.2.100
leaf2# show nve peers
leaf2# show mac address-table vlan 1000
leaf2# show ip arp vrf all

Read the checklist top to bottom and stop at the first command that returns something unexpected. VNI state should show Up on every leaf, peers should list every VTEP in the local fabric, the BGW sessions should be Established in both directions, and the type-2 and type-5 route counts should be symmetric. When they are not, the counters on the BGW's inter-site interface tell you whether traffic is being dropped by policy or never arriving.

Troubleshooting Decision Tree

Use this ordering to cut the search space quickly:

Source host cannot ARP its gateway → local problem, stop the multisite hunt.
Gateway ARP works, remote MAC absent from the local table → the type-2 route is not arriving. Check the BGW's advertised-routes and the RT import.
Remote MAC present, no type-5 route for the remote subnet → the routed path is missing. Check the L3 VNI and its RT on both BGWs.
Both present, ping fails both ways → capture at the BGW. One-way delivery means next-hop or route-target asymmetry.
Ping works, larger traffic stalls → MTU. Test with a stepped DF-bit ping before blaming the fabric.

Layer 2 or Layer 3 Stretched? Choose Deliberately

Every multisite design is a trade between simplicity and blast radius. Stretching a Layer 2 VNI across sites gives you one subnet and painless live migration, but it also extends the failure domain: a broadcast storm or an ARP flood crosses the inter-site link and affects both data centers. A routed multisite - separate subnets per site, type-5 routes exchanged between BGWs - keeps the failure domains apart and is far easier to troubleshoot, because a ping failing across a routed boundary points at a route or a policy rather than at an opaque bridge domain.

If you must stretch Layer 2, limit it: keep the stretched VNI's subnet small, rate-limit broadcast and unknown unicast on the inter-site link, and never stretch a subnet that carries a protocol with aggressive broadcast behaviour. Troubleshooting a stretched Layer 2 problem across sites is considerably harder than the equivalent routed problem, and the extra command surface is where most of the time goes.

Underlay Health: The Check Everyone Skips

Everything above assumes the underlay is healthy, and in multisite designs that assumption is worth testing rather than trusting. The VXLAN tunnel is only as good as the routed path beneath it, and a single congested or flapping inter-site link will produce EVPN symptoms that look like control-plane faults. Verify the underlay before you spend an afternoon reading BGP tables.

BGW1# show interface ethernet 1/1 counters errors
BGW1# show ip route 10.10.10.5
BGW1# traceroute 10.10.10.5 source 10.10.10.1
BGW1# show ip ospf neighbors
BGW1# show bgp ipv4 unicast summary

Look for incrementing CRC or input errors on the inter-site ports, a route to the remote VTEP loopback that has changed next hop recently, and any IGP adjacency that has flapped within the last hour. Routing protocol log entries with a timestamp inside the outage window are the strongest evidence there is, and they cost nothing to check. If the underlay shows even minor instability, fix that first: chasing an MTU or route-target theory across a link that is dropping packets at random wastes more time than any other mistake in this article.

Finally, confirm that the inter-site link's MTU accounts for the VXLAN overhead. Fifty bytes of outer headers on top of a 1500-byte inner frame does not fit a 1500-byte transit link, and the resulting behaviour - small pings succeed, real application traffic stalls - is one of the most frequently misdiagnosed faults in stretched data centers. Raise the underlay MTU or clamp TCP MSS at the border, and document which of the two you chose.