VXLAN EVPN Multi-Site: Border Gateway Design Guide

A single VXLAN EVPN fabric is limited by the number of VTEPs one control plane can carry and by the fact that any control-plane fault is now a fault in every site. Multi-site solves both: each site keeps its own spine layer as an independent fault domain, and a small set of border gateways (BGWs) stitches the sites together over a routed backbone. The design decision that confuses most teams is why the BGWs deliberately do not leak remote EVPN routes from one site into another. This article explains the architecture, the route handling, the failure behaviour of the BGW pair, and the configuration skeleton.

Why not just stretch one fabric?

• Fault isolation. A control-plane problem, a BUM storm or a misbehaving VTEP stays inside its site.
• Scale. Each site has its own BGP EVPN control plane; the BGWs summarise, so the number of routes a leaf must carry stays bounded.
• Backbone independence. The inter-site transport is just routed IP — MPLS, DWDM, a service provider WAN or a cloud interconnect. No VXLAN over VXLAN requirement.

There is a fourth reason that only becomes obvious during an incident: blast radius. In a stretched fabric, a duplicated ESI, a leaked route target or a VTEP with a wrong VNI has fabric-wide consequences, and the failure domain of every tenant is the whole network. Multi-site gives you a natural administrative boundary. Tenants stay attached to their own site's leaf layer, the BGW is the only place where policy is enforced, and a change in one site is reviewed and tested in that site's context.

The Multi-Site Vocabulary

Four terms appear in every multi-site design document, and they need to be separated before any configuration makes sense:

• PIP (primary IP VTEP). The VTEP address inside the site — normally a dedicated loopback used as the NVE source interface. Every leaf and BGW in the site speaks to it.
• VIP (virtual IP VTEP) or anycast VTEP. A second, distinct loopback advertised as the multi-site border-gateway interface. All BGWs of a site share the same VIP, so every other site has exactly one address to send inter-site traffic to.
• Site ID. A value configured on every BGW of the site and, critically, identical on all of them. It is what makes the BGWs of one site recognisable as a set, and a mismatched site ID is a silent misconfiguration that produces two apparent multi-site domains.
• Inter-site network (ISN). The routed transport between sites. It carries only IP, plus the BGP EVPN sessions between BGWs; it does not carry VXLAN between leaf tiers.

! the two loopbacks must be different addresses
interface loopback0
  description "Site VTEP source (PIP)"
  ip address 10.31.1.1/32

interface loopback100
  description "Multi-site VIP (anycast VTEP)"
  ip address 10.100.100.1/32

Both loopbacks must be advertised into the site's underlay IGP so that every leaf can reach the VIP with equal cost through either BGW. The VIP is not a BGP next hop for local routes; it is the address that remote sites use as the destination for inter-site traffic arriving in the local site.

How the border gateway handles routes

Each BGW is a VTEP in its local site and a plain IP speaker on the backbone. It advertises a local anycast VTEP address into every site so that all sites agree on the same BGW ingress point. For each EVPN route type the treatment differs:

• Type-2 (MAC/IP) — local MAC advertisement is re-originated into the local site with the BGW’s own VTEP as next hop. Remote-site Type-2 routes are not flooded across other sites; host reachability between sites is handled by the BGW pair.
• Type-3 (IMET) — the BGW advertises itself as an inclusive multicast endpoint so that BUM traffic reaches the remote site through the BGW rather than directly.
• Type-5 (IP prefix) — used for inter-subnet and external reachability. Type-5 keeps the routing architecture clean: host prefixes are advertised as IP prefixes rather than MAC/IP pairs, which is what allows summarisation at the BGW.

This is intentional: the standard multi-site design prevents BGWs from forwarding EVPN routes between remote sites in order to isolate failure domains and keep the route count manageable.

Two practical consequences follow from that rule. First, because Type-2 anycast information is contained within the fabric, a route-target policy problem is contained too: a missing route target breaks inter-site reachability loudly and locally instead of invisibly modifying the whole network. Second, when the tenant VRFs are extended across the ISN, route targets must be defined explicitly on the BGW for each VRF it extends — a requirement that is enforced in current NX-OS releases but was not in early ones, where intra-site route targets could be kept across the site boundary unintentionally. Note also the ASN handling on the inter-site EVPN session: a neighbour configured as fabric-external needs the rewrite-evpn-rt-asn capability so that route targets are rewritten as they cross the site boundary, which keeps a single route-target scheme usable across sites with different AS numbers.

Inter-Site Control Plane Design

The BGP design between BGWs is deliberately simple, and keeping it simple is what makes the failure behaviour predictable:

• One ASN per site. Site-autonomous-system design makes a leaked route obvious in the AS path and keeps the ISN policy expressible as an ordinary inter-AS policy.
• Sessions between BGWs, not between sites. Each BGW peers with the BGWs of the remote sites over the ISN, sourcing the session from the VIP loopback with a multihop configuration.
• Only the L2VPN EVPN and IPv4 unicast address families. The ISN carries the tenant VRFs and the VIP reachability; nothing else needs to cross.
• Route-server topology where scale demands it. A route server on the ISN removes the need for a full mesh of BGW-to-BGW sessions while leaving the EVPN semantics unchanged.

NX-OS configuration skeleton

# Features
feature bgp
feature vn-segment-vlan-based
feature nv overlay

# Local anycast VTEP and multi-site border gateway
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  multisite border-gateway 65001
  multisite border-gateway interface loopback100
  multisite delay-restore 180

# Verify border-gateway state and route counting
show nve multisite border-gateway status
show bgp l2vpn evpn summary
show bgp l2vpn evpn
show nve vni
show l2route evpn mac all

multisite delay-restore keeps VXLAN tunnels to remote sites down for a configurable period after a reload so that the local site converges before remote traffic arrives. On platforms where the backbone VRF is separate, the BGW needs a dedicated inter-site VRF with its own BGP session to the peer BGWs.

The delay-restore parameter is not a cosmetic timer; it is the guard against the most common reload failure in multi-site, which is a BGW advertising its VIP into the ISN before its own site's underlay and overlay have converged. A value of 300 seconds is a common starting point on large fabrics, and it should be tuned against the measured convergence time of the site rather than guessed. On platforms that support it, there is also a separate delay-restore for SVIs, which is useful when the SVI and VNI scale is high.

Failure Behaviour of the Border Gateway Pair

Multi-site is designed so that losing one BGW is a data-plane event rather than a control-plane one. Two mechanisms implement that:

• Fabric tracking. Each BGW monitors its own site-internal interfaces and withdraws the VIP from the site and the ISN when its local fabric path fails. This prevents the failure mode where a BGW is up, has an ISN path, and silently black-holes traffic because it cannot reach the leaves.
• DCI tracking. The same logic applied to the inter-site path: if the BGW loses its link to the ISN, it withdraws from the multi-site role so remote sites use the other BGW.

With anycast BGW the traffic is load-shared across the BGWs of the site, so a single BGW failure removes half the inter-site capacity but does not require a new anycast address to be learned. The trigger delay matters here: because the VIP is withdrawn and re-advertised as an anycast prefix, a path that flaps produces repeated BGP updates toward every remote site. Multi-hop BFD between BGWs plus a next-hop trigger delay is the standard remedy, and it keeps the withdrawal fast while preventing the route from bouncing.

An ES-like consistency point is also worth designing in: all BGWs of a site must share the site ID, the VIP address, and the same set of extended VRFs. A BGW that is missing one tenant VRF does not fail loudly. It attracts inter-site traffic with the anycast VIP for a VRF it cannot map, and the packets are dropped. This is the multi-site equivalent of a missing VLAN, and the only reliable detection is comparing configuration across the BGW set.

Design rules that keep the fabric predictable

• Use the same anycast VTEP IP on both BGWs of a site pair, and make sure the backbone IGP provides equal-cost reachability to it.
• Keep tenants consistent: a VNI that exists in site A but not site B produces silent blackholes, because Type-5 or Type-2 routes are accepted but the VNI cannot be mapped.
• Decide explicitly where the Layer 3 boundary is. Either every site does its own anycast distributed gateway and Type-5 is used only for external prefixes, or the BGWs are the inter-site routing points — mixing both creates asymmetric routing that is painful to debug.
• Test with a real control-plane failure: shut one BGW and verify that remote sites reconverge to the remaining anycast VTEP within your target window.
• Keep the NVE source loopback and the multi-site VIP loopback distinct, and document which one is used for what, because a maintenance window that moves the wrong address takes down both local and inter-site forwarding.
• Inspect the underlay MTU before adding a site: VXLAN adds encapsulation overhead, and an inter-site path that carries jumbo frames locally but fragments on the WAN produces failures that look like application problems.

Troubleshooting Checklist

show nve multisite border-gateway status
show nve interface nve1
show bgp l2vpn evpn summary
show bgp l2vpn evpn neighbors 10.32.1.1
show nve vni
show l2route evpn mac all | include <mac>
show l2route evpn imet
show ip route 10.100.100.1/32

Work the list in order. If the border-gateway status is not confirmed, nothing else will behave correctly. If the BGP EVPN session to the remote BGW is up but the Type-5 routes for a tenant are missing, the problem is route-target or VRF extension, not the ISN. If the routes are present but traffic still fails, check the VNI mapping on both sides and the MTU along the inter-site path — in that order.

Related reading: Nexus 9000 VXLAN BGP EVPN design, Junos EVPN-VXLAN CRB fabric configuration and Huawei VXLAN distributed gateway with BGP EVPN.

原文链接：https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html