Cisco Nexus 9000 VXLAN BGP EVPN Fabric Design Guide

A VXLAN BGP EVPN fabric is more than a pile of VLAN-to-VNI mappings — it's a set of design decisions about CLOS topology, device roles, underlay routing, and IP addressing that determine whether the fabric scales cleanly or becomes an operational nightmare. Cisco's fundamental design and implementation guide for the Nexus 9000 explains every decision point. This article distills the essentials: 3-stage vs 5-stage CLOS, the four leaf roles, what the spine should and shouldn't do, and the IP addressing plan for leaf and spine nodes.

CLOS Topology: 3-Stage and 5-Stage

The 3-stage CLOS has leaf and spine layers: users, applications, L4-L7 services, and external networks attach to leaves; spines connect to every leaf and transit all inter-leaf traffic. Adding leaves scales out server capacity; adding spines scales bandwidth and reduces oversubscription. The 5-stage CLOS adds a super-spine layer that interconnects spines across sites — useful for compartmentalizing very large fabrics or as a data center interconnect (DCI) transport.

Leaf Roles in a VXLAN Fabric

• Leaf (VTEP): where virtual or physical servers attach to the VXLAN network.
• Border Leaf: VTEP attaching to edge/WAN routers; the VXLAN-to-VRF-lite handoff point for north-south traffic. At scale, use a dedicated border leaf — a shared one becomes a chokepoint for both east-west and north-south traffic and consumes TCAM for external network programming.
• Border Gateway: VTEP used as a DCI node in multisite fabrics.
• Service Leaf: attaches L4-L7 devices such as firewalls, load balancers, and IDS/IPS.

A spine, by contrast, is a pure transit device: it learns overlay routes from every leaf and redistributes them, runs the MP-BGP EVPN route reflector, and (in multicast underlays) can act as the rendezvous point. Keep it simple — dedicated roles mean simpler, safer configurations.

Underlay and IP Addressing Plan

Leaf nodes need four classes of IP addresses: fabric interface IPs, loopbacks, L2 VNI gateway (SVI) IPs, and infrastructure-service loopbacks (DHCP relay / VXLAN OAM). Best practices:

• Fabric interfaces: use /30 or /31 point-to-point addressing, or better, IP unnumbered — interfaces borrow the loopback0 address, cutting the number of transit IPs and simplifying automation.
• Loopbacks: at least one for underlay routing (router ID) and one for the NVE interface; vPC fabric peering may need a dedicated loopback for the virtual peer link.
• L2 VNI gateway IP: the SVI IP per VLAN, acting as the distributed anycast gateway for hosts.
• Infrastructure loopback in a VRF: a unique per-leaf address for DHCP/OAM so responses can be attributed to the correct VTEP — you can't use the shared anycast SVI IPs for this.

For implementation on other vendors, compare with our Junos EVPN-VXLAN CRB configuration article, the HPE Aruba Fabric Composer guide, and the spine-leaf vs three-tier design overview.

原文链接：https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-vxlan-bgp-evpn-design-and-implementation-guide.html