Cisco Nexus 9000 VXLAN BGP EVPN Fabric Design Guide

A VXLAN BGP EVPN fabric is more than a pile of VLAN-to-VNI mappings — it's a set of design decisions about CLOS topology, device roles, underlay routing, and IP addressing that determine whether the fabric scales cleanly or becomes an operational nightmare. Cisco's fundamental design and implementation guide for the Nexus 9000 explains every decision point. This article distills the essentials: 3-stage vs 5-stage CLOS, the four leaf roles, what the spine should and shouldn't do, and the IP addressing plan for leaf and spine nodes.

CLOS Topology: 3-Stage and 5-Stage

The 3-stage CLOS has leaf and spine layers: users, applications, L4-L7 services, and external networks attach to leaves; spines connect to every leaf and transit all inter-leaf traffic. Adding leaves scales out server capacity; adding spines scales bandwidth and reduces oversubscription. The 5-stage CLOS adds a super-spine layer that interconnects spines across sites — useful for compartmentalizing very large fabrics or as a data center interconnect (DCI) transport.

Oversubscription is the number that drives the choice. A leaf with 48 downlinks at 25 Gbps and 6 uplinks at 100 Gbps is 1200:600, or 2:1 oversubscribed — perfectly reasonable for general-purpose server access. Cutting to 4 uplinks takes it to 3:1, which is fine for storage-light workloads and painful for a Hadoop or GPU cluster. Every spine you add widens the uplink pool without touching the leaf configuration, which is why spine scale-out is described as nearly free in a CLOS.

Leaf Roles in a VXLAN Fabric

• Leaf (VTEP): where virtual or physical servers attach to the VXLAN network.
• Border Leaf: VTEP attaching to edge/WAN routers; the VXLAN-to-VRF-lite handoff point for north-south traffic. At scale, use a dedicated border leaf — a shared one becomes a chokepoint for both east-west and north-south traffic and consumes TCAM for external network programming.
• Border Gateway: VTEP used as a DCI node in multisite fabrics.
• Service Leaf: attaches L4-L7 devices such as firewalls, load balancers, and IDS/IPS.

A spine, by contrast, is a pure transit device: it learns overlay routes from every leaf and redistributes them, runs the MP-BGP EVPN route reflector, and (in multicast underlays) can act as the rendezvous point. Keep it simple — dedicated roles mean simpler, safer configurations.

Two rules follow from that role split. First, never attach endpoints to a spine; the moment one does, the fabric has a path that bypasses the leaf layer and every troubleshooting assumption breaks. Second, if the spine is doing route reflection, make sure the spine pair runs the same cluster ID so that a route reflected by one spine is recognised as a duplicate by the other rather than a second path — that detail alone prevents most transient routing loops during spine maintenance.

Underlay and IP Addressing Plan

Leaf nodes need four classes of IP addresses: fabric interface IPs, loopbacks, L2 VNI gateway (SVI) IPs, and infrastructure-service loopbacks (DHCP relay / VXLAN OAM). Best practices:

• Fabric interfaces: use /30 or /31 point-to-point addressing, or better, IP unnumbered — interfaces borrow the loopback0 address, cutting the number of transit IPs and simplifying automation.
• Loopbacks: at least one for underlay routing (router ID) and one for the NVE interface; vPC fabric peering may need a dedicated loopback for the virtual peer link.
• L2 VNI gateway IP: the SVI IP per VLAN, acting as the distributed anycast gateway for hosts.
• Infrastructure loopback in a VRF: a unique per-leaf address for DHCP/OAM so responses can be attributed to the correct VTEP — you can't use the shared anycast SVI IPs for this.

Write the plan down before the first switch is racked. A fabric built by hand without an addressing document is nearly impossible to extend six months later, because IP unnumbered hides the link addresses you would otherwise be able to infer.

Choosing the Underlay Protocol

The underlay only needs to do one thing well: provide loopback-to-loopback reachability with fast, equal-cost multipathing. Three options dominate.

• OSPF — the most common choice in Cisco fabrics. Simple, well understood, and the leaf-spine topology is just a set of point-to-point links in one area. Tune the timers or run BFD for fast failure detection.
• IS-IS — preferred in very large fabrics and by service providers. Scales better, fewer knobs, and behaves predictably during flooding.
• eBGP — one AS per leaf or per rack. Chosen when the operational model is "BGP everywhere," and it makes the underlay a natural extension of the overlay's policy model, but it needs more configuration discipline around next-hop handling.

Whichever you pick, enable maximum-paths (ECMP) and pair it with BFD. A fabric where the underlay takes 40 seconds to notice a dead spine uplink has no chance of meeting an application-level availability target, no matter how good the overlay is.

Overlay: BGP EVPN Route Types

The overlay is where EVPN earns its keep over flooding-and-learning. BGP carries a small number of route types, and knowing them makes show bgp l2vpn evpn output readable instead of intimidating.

• Type 2 (MAC/IP Advertisement) — the workhorse. Advertises a host's MAC, its IP, and the VTEP that owns it. This is what removes the need for data-plane MAC learning across the fabric and enables ARP suppression.
• Type 3 (Inclusive Multicast Ethernet Tag) — signals that a VTEP wants to receive BUM (broadcast, unknown unicast, multicast) traffic for a given VNI. Without a Type 3 route for a VNI, a VTEP will not join that VNI's flood list.
• Type 5 (IP Prefix) — advertises a routable IP prefix into the EVPN table, used for external connectivity and for L3 handoff to a border leaf.

The spines act as EVPN route reflectors, which keeps full-mesh iBGP sessions off the leaf layer — with 40 leaves, full mesh means 780 sessions; route reflection means 40. Two spines as a reflector pair give redundancy without doubling the configuration burden.

Anycast Gateway and vPC

A distributed anycast gateway gives every leaf the same SVI IP and the same virtual MAC for a given VLAN, so a host can move anywhere in the fabric and keep its default gateway. fabric forwarding anycast-gateway-mac 0200.0000.1234 plus the same per-SVI address on every leaf is the whole trick, and it eliminates the classic first-hop redundancy protocols from the design entirely.

For dual-homed servers, vPC provides the multi-chassis link aggregation. In modern fabrics the peer link can be carried over the VXLAN overlay itself (vPC fabric peering), which frees the physical ports that a dedicated peer link would consume — but it needs the dedicated peer-link loopback mentioned in the addressing plan and a consistent MTU end to end.

VLAN/VNI Planning and MTU

• Keep VLAN IDs and VNI numbers identical where possible (VLAN 101 maps to VNI 10101) so no one has to translate in their head at 3 a.m.
• Reserve a documented block for L2 VNIs and another for L3 VNIs, and record which VLANs get an SVI.
• Set the fabric-wide MTU to at least 9216 on every link in the path, spine and leaf alike. VXLAN adds 50 bytes of overhead, and a single 1500-byte link in the path produces the classic "large pings fail, small ones work" symptom.
• Decide multicast vs ingress replication for BUM traffic early. Ingress replication is simpler and fine for smaller fabrics; multicast underlays scale better at large VNI counts but add PIM to every device.

Putting It Together: A Reference Leaf Configuration

The design decisions above translate into a fairly short configuration. This is the shape of a leaf's underlay, overlay and anycast gateway on NX-OS:

! Underlay: loopback and point-to-point fabric links
interface loopback0
  ip address 10.0.0.11/32
interface Ethernet1/1
  no switchport
  mtu 9216
  ip unnumbered loopback0
  ip router ospf UNDERLAY area 0.0.0.0
  ip ospf network point-to-point

! Overlay: NVE interface bound to the underlay loopback
interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10101
    ingress-replication protocol bgp
  member vni 50101
    mroute 239.1.1.1

! BGP EVPN as the overlay control plane
router bgp 65001
  router-id 10.0.0.11
  address-family l2vpn evpn
    neighbor 10.0.0.1
      remote-as 65001
      update-source loopback0
      send-community extended

! Anycast gateway SVI
interface Vlan101
  no shutdown
  vrf member TENANT-A
  ip address 192.0.2.1/24
  fabric forwarding mode anycast-gateway

Every line maps to a decision made earlier: ip unnumbered comes from the addressing plan, host-reachability protocol bgp from the choice to use EVPN instead of flood-and-learn, send-community extended because EVPN route targets are extended communities, and the identical SVI address on every leaf is what makes the distributed anycast gateway work. The spine side is deliberately simpler — underlay routing plus an EVPN address-family acting as a route reflector, with no VNI membership and no SVIs. That asymmetry is a feature: the more the spine resembles a plain transit router, the less there is to misconfigure during a maintenance window.

Capacity and Failure Domain Planning

Two questions decide whether the fabric survives its first bad day. First, what happens when one spine dies? With N spines, losing one removes roughly 1/N of the fabric's bisection bandwidth, and every leaf must retain usable ECMP paths to the survivors. Two spines is the minimum for redundancy and four is a common production choice; a "pair" where each leaf has only a single uplink to one of them is not a fabric, it is a triangle waiting to fail.

Second, what happens when one leaf dies? Its VNIs go with it, and every host attached to it loses connectivity no matter how good the overlay is. Multi-homing hosts to a vPC pair, or designing the application to tolerate a leaf loss, is the only real answer. Decide the failure-domain boundary for the overlay early as well: if a spine is simultaneously the underlay router and the EVPN route reflector, losing it removes both functions in one event, which is acceptable only when a second spine carries the same roles. Writing down these two numbers — spines per layer, and leaf-loss tolerance — turns the fabric's resilience into a design property instead of an accident.

Multisite Considerations

When two fabrics must be joined, the multisite design introduces a border gateway per site and a BGP EVPN session carrying the overlay between them. Keep the site-local VLAN and VNI numbering consistent between sites, keep the EVPN route reflectors site-local, and be explicit about which VNIs span sites — every stretched subnet is a stretch of failure domain as well.

For implementation on other vendors, compare with our Junos EVPN-VXLAN CRB configuration article, the HPE Aruba Fabric Composer guide, and the spine-leaf vs three-tier design overview. For a full multisite build, see VXLAN EVPN multi-site border gateway design, and avoid the most common implementation failure with VXLAN MTU and fragmentation: the 50-byte tax.

原文链接：https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-vxlan-bgp-evpn-design-and-implementation-guide.html