VXLAN EVPN Multi-Site: Border Gateway Design Guide

A single VXLAN EVPN fabric is limited by the number of VTEPs one control plane can carry and by the fact that any control-plane fault is now a fault in every site. Multi-site solves both: each site keeps its own spine layer as an independent fault domain, and a small set of border gateways (BGWs) stitches the sites together over a routed backbone. The design decision that confuses most teams is why the BGWs deliberately do not leak remote EVPN routes from one site into another. This article explains the architecture, the route handling, and the configuration skeleton.

Why not just stretch one fabric?

• Fault isolation. A control-plane problem, a BUM storm or a misbehaving VTEP stays inside its site.
• Scale. Each site has its own BGP EVPN control plane; the BGWs summarise, so the number of routes a leaf must carry stays bounded.
• Backbone independence. The inter-site transport is just routed IP — MPLS, DWDM, a service provider WAN or a cloud interconnect. No VXLAN over VXLAN requirement.

How the border gateway handles routes

Each BGW is a VTEP in its local site and a plain IP speaker on the backbone. It advertises a local anycast VTEP address into every site so that all sites agree on the same BGW ingress point. For each EVPN route type the treatment differs:

• Type-2 (MAC/IP) — local MAC advertisement is re-originated into the local site with the BGW’s own VTEP as next hop. Remote-site Type-2 routes are not flooded across other sites; host reachability between sites is handled by the BGW pair.
• Type-3 (IMET) — the BGW advertises itself as an inclusive multicast endpoint so that BUM traffic reaches the remote site through the BGW rather than directly.
• Type-5 (IP prefix) — used for inter-subnet and external reachability. Type-5 keeps the routing architecture clean: host prefixes are advertised as IP prefixes rather than MAC/IP pairs, which is what allows summarisation at the BGW.

This is intentional: the standard multi-site design prevents BGWs from forwarding EVPN routes between remote sites in order to isolate failure domains and keep the route count manageable.

NX-OS configuration skeleton

# Features
feature bgp
feature vn-segment-vlan-based
feature nv overlay

# Local anycast VTEP and multi-site border gateway
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  multisite border-gateway 65001
  multisite delay-restore 180

# Verify border-gateway state and route counting
show nve multisite border-gateway status
show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve vni
show l2route evpn mac all

multisite delay-restore keeps VXLAN tunnels to remote sites down for a configurable period after a reload so that the local site converges before remote traffic arrives. On platforms where the backbone VRF is separate, the BGW needs a dedicated inter-site VRF with its own BGP session to the peer BGWs.

Design rules that keep the fabric predictable

• Use the same anycast VTEP IP on both BGWs of a site pair, and make sure the backbone IGP provides equal-cost reachability to it.
• Keep tenants consistent: a VNI that exists in site A but not site B produces silent blackholes, because Type-5 or Type-2 routes are accepted but the VNI cannot be mapped.
• Decide explicitly where the Layer 3 boundary is. Either every site does its own anycast distributed gateway and Type-5 is used only for external prefixes, or the BGWs are the inter-site routing points — mixing both creates asymmetric routing that is painful to debug.
• Test with a real control-plane failure: shut one BGW and verify that remote sites reconverge to the remaining anycast VTEP within your target window.

Related reading: Nexus 9000 VXLAN BGP EVPN design, Junos EVPN-VXLAN CRB fabric configuration and Huawei VXLAN distributed gateway with BGP EVPN.

原文链接：https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html