Spine-Leaf vs Traditional 3-Tier: Design and Migration Guide

Two decades of three-tier data center design (core, aggregation, access) is giving way to spine-leaf for a simple reason: modern east-west traffic — virtualization, Kubernetes, distributed storage and AI clusters — cannot survive the blocked links and aggregation bottlenecks of the legacy model. This guide breaks down the mechanics of a spine-leaf fabric, contrasts it against three-tier with concrete numbers, and outlines how to migrate an existing brownfield network without redesigning it from scratch.

What follows is written for the engineer who has to produce a design document, a bill of materials and a migration plan - not just a topology diagram. We will cover the Clos maths, the oversubscription calculations you will be asked to defend, the underlay and overlay configuration, the failure domains that change when you flatten the network, and the phased cut-over that keeps the business running while you do it.

Spine-Leaf: The Clos Two-Tier Model

Spine-leaf flattens the topology to two tiers based on the Clos network topology formalized by Charles Clos in 1952. Every leaf switch (ToR/EoR, where servers and storage attach) connects to every spine switch, and the unbreakable rule is: every leaf to every spine; never spine-to-spine or leaf-to-leaf. The result is mathematically predictable — any server is exactly two hops away from any other server.

The terminology is worth nailing down because it drives the design. A leaf is where endpoints attach; it is the boundary between the server world and the fabric. A spine is an aggregation point that connects only to leaves and carries no endpoint traffic of its own. A leaf may be a ToR (top of rack, one pair per rack) or an EoR (end of row, cabling a whole row back to two switches). ToR wins on cabling volume and failure granularity; EoR wins when rack space and switch count matter more than cable runs.

The two-hop property is the point. In a three-tier network, two hosts in different racks can be five or six hops apart through aggregation and core, each hop a place for a blocked link, a filtering policy or a microburst queue to live. In a two-tier Clos, the path is leaf -> spine -> leaf, always, and the fabric is symmetric by construction. Adding a spine adds bandwidth for every leaf simultaneously, without touching the leaf-to-leaf path length.

Port count, or radix, sets the ceiling. If each leaf has 32 uplink ports, you can have at most 32 spines; if you have 32 spines, each leaf can use all 32 uplinks in an ECMP bundle. When you need more leafs than the radix allows, you do not widen the spine layer - you build a third tier Clos (leaf, spine, superspine), which is why very large fabrics eventually look like three tiers again, but with a completely different forwarding model: every tier is routed, every tier uses ECMP, and nothing is blocked.

Why STP Kills Three-Tier Performance

Three-tier networks rely on Spanning Tree Protocol to prevent loops, and STP actively blocks redundant links. Two 40G uplinks provisioned for redundancy means you pay for 80Gbps but use 40Gbps, and a link failure triggers seconds of STP convergence. Spine-leaf pushes Layer 3 routing to the ToR and uses ECMP instead: a leaf with four 100G uplinks to four spines load-balances flows across all four links simultaneously, achieving 100% utilization and sub-millisecond failover.

The blocking cost compounds at the aggregation layer, which is exactly where three-tier concentrates traffic. Access-to-aggregation links are traditionally Layer 2, so half of them sit idle; aggregation-to-core links are often Layer 3 with two paths, so ECMP can use both but there are only two. Every rack that was dual-homed for redundancy contributed to a topology where roughly 50% of the purchased bandwidth was reserved for failure rather than carrying traffic. In a spine-leaf fabric the same cabling is all active at once: four uplinks mean four times the usable capacity, not two times with one standing by.

Convergence differs just as sharply. An STP topology change, even with RSTP, involves a proposal/agreement round, MAC table flushes, and a ripple through every downstream switch for a few hundred milliseconds to a few seconds. A routed fabric with BFD and tuned IGP/BGP timers fails over in tens of milliseconds, and because every path is equal-cost, there is no reconvergence at all for a single link loss in a four-way bundle - the hardware simply removes the link from the ECMP set.

Overlays: VXLAN and BGP EVPN

Routing at the leaf breaks Layer 2 adjacency for VM mobility, so spine-leaf fabrics use VXLAN (MAC-in-UDP, 24-bit VNI = 16.7 million segments) with BGP EVPN as the control plane. Instead of flood-and-learn, leaves advertise MAC and IP addresses via BGP, acting as a distributed synchronized database that updates instantly when a VM migrates.

The underlay is a plain routed fabric: IP unnumbered or /31 links from each leaf to each spine, a loopback per leaf and spine as the router ID and VTEP source, and OSPF, ISIS or eBGP carrying the loopbacks and uplinks. It should stay boring and minimal - no tenant addressing, no policy. Every leaf and spine that will terminate VXLAN needs a /32 loopback reachable from every other fabric device.

The overlay carries tenant VLANs and segments. Each leaf is a VTEP, encapsulating frames in UDP 4789 with a VNI that identifies the tenant segment. BGP EVPN (address family l2vpn evpn) advertises type-2 routes for MAC and ARP entries and type-3 routes for the VTEP flood list, so a leaf knows which remote VTEP owns a MAC before the first packet is sent. Integrated Routing and Bridging (IRB) with a distributed anycast gateway puts the same gateway IP and MAC on every leaf, so a VM keeps its default gateway wherever it moves - this is what finally makes Layer 2 stretching unnecessary for mobility.

An example of the leaf-side essentials on a Cisco Nexus 9000:

feature bgp
feature interface-vlan
feature nv overlay
feature vn-segment-vlan-based

vlan 100
  vn-segment 10100

interface Vlan100
  no shutdown
  ip address 10.100.0.1/24
  fabric forwarding mode anycast-gateway

interface nve1
  no shutdown
  source-interface loopback0
  host-reachability protocol bgp
  member vni 10100
    ingress-replication protocol bgp

And the EVPN peering to the spine, which is an L2VPN-only session:

router bgp 65001
  neighbor 10.255.0.1 remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
  neighbor 10.255.0.2 remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
  address-family l2vpn evpn
    vni 10100 l2
      rd auto
      route-target both auto

Calculating Oversubscription

Oversubscription = (server ports × speed) ÷ (uplink ports × speed). Example: a leaf with forty-eight 10G server ports (480 Gbps) and four 100G uplinks (400 Gbps) gives 480:400 = 1.2:1. General enterprise virtualization typically runs 3:1; AI/GPU clusters and NVMe-oF demand 1:1 non-blocking.

The ratio you are asked to defend is usually east-west, not north-south, and the arithmetic is unforgiving once you count storage replication. A few worked cases:

Two refinements matter. First, the ratio applies to the fabric as a whole only if every leaf has the same ratio; mixing 3:1 and 1.2:1 leaves is a capacity trap. Second, spine oversubscription is the ratio of all leaf uplinks to all spine downlinks summed across the fabric - since each spine connects to every leaf, that number is normally 1:1 and the only oversubscription left in the design lives at the leaf. If you need a real TCO argument, tell the storage team their replication traffic is 40% of the leaf downlink capacity and it all goes east-west through the spines.

Underlay Routing and ECMP

The underlay should be as simple as you can make it while still being deterministic. A common choice is unnumbered point-to-point links with eBGP per leaf-spine pair, which scales without any IP planning for links and gives you a clean failure domain: each leaf-spine adjacency is independent, and losing one link only shrinks that leaf's ECMP set.

router bgp 65001
  router-id 10.255.0.11
  maximum-paths 8
  maximum-paths ibgp 8
  neighbor 10.255.0.1 remote-as 65001
    update-source loopback0
  neighbor 10.255.0.2 remote-as 65001
    update-source loopback0
  address-family ipv4 unicast
    redistribute direct route-map LOOPBACKS
  address-family l2vpn evpn
    neighbor 10.255.0.1 activate
    neighbor 10.255.0.2 activate

Verify that the fabric actually installs multiple paths, because a single-path underlay silently halves your bandwidth:

show ip route 10.255.0.99
show ip route summary
show bgp l2vpn evpn summary
show nve peers
show nve vni

Also check that ECMP hashing is not collapsing flows. Servers with a single TCP flow to a single destination will take one path, and that is expected; the fabric is designed for flow-level, not packet-level, spraying. If a specific elephant flow saturates one uplink, the fix is on the server side (multiple flows, RSS, or NIC-based dynamic load balancing), not in the fabric.

Multi-Tier Clos and Fabric Scale

A two-tier Clos is limited by spine radix: with 32-port spines you can build 32 spines and, if each leaf uses all 32 uplinks, as many leaves as your spine ports allow - in practice each spine supports N leaf ports, so 32 spines × 32 leaf-facing ports is 1024 leaf connections, or 512 leafs at two links each. When the leaf count exceeds that, you add a superspine tier and the fabric becomes a three-stage Clos again, but every stage is routed and every stage is ECMP.

Three-stage Clos is not a return to three-tier design. In three-tier, the aggregation layer is a Layer 2 conduit that blocks redundant links; in three-stage Clos, the superspines are routed, all links active, and the path length is fixed at four hops regardless of which racks are talking. The design rules are the same - full mesh between adjacent tiers, never tier-crossing links, never a host attached to a spine or superspine.

Brownfield Migration Strategy

A pragmatic migration keeps the existing access tier while building a spine fabric above it: deploy the spine pair and new leaf switches, migrate workloads rack by rack to the leaf tier, run VXLAN/BGP EVPN between leaf and spine, and gradually decommission the aggregation layer. Stateful services (firewalls, load balancers) move to dedicated service leaves with VRF leaking and policy-based routing. Cisco Nexus 9000, Arista 7050X/7800R and Juniper QFX are the typical hardware platforms.

The full sequence, which usually spans several change windows:

1. Baseline and map. Document every VLAN, subnet and default gateway currently anchored at the aggregation layer, and every service (firewall, load balancer, proxy) that sits inline. This list is your migration checklist and it is almost always longer than anyone expects.
2. Build the underlay. Install the spine pair, cable and route the new leafs to the spines only, and confirm loopback reachability with show ip route before any tenant configuration exists. Nothing else touches the fabric yet.
3. Stand up the overlay. Configure NVE, VNIs and EVPN peering on the new leafs; verify show nve peers shows all spines and no VNI is down. Test with two lab hosts on two different leafs in the same VNI before touching production.
4. Migrate rack by rack. Move one rack at a time: dual-home the hosts to the new leaf pair, move the VLANs into the VNI, and shift the default gateway to the anycast address. Because the anycast gateway is the same everywhere, this is an address-preserving move from the server's point of view.
5. Extend to the legacy aggregation. Where an old aggregation block still owns subnets, connect it to the new spine or a border leaf and redistribute, so the legacy and new worlds route to each other during the transition. This is a temporary bridge, not a permanent design.
6. Move services to service leaves. Firewalls, load balancers and inline inspection appliances attach to dedicated service leafs, with VRFs and route leaking defining which tenants traverse them. Avoid drawing a service device into the middle of the fabric data path.
7. Decommission. Once no VLAN terminates at aggregation and no interface carries production traffic, remove the devices and reclaim rack space, power and maintenance contracts.

The most common failure in this process is trying to migrate the IP addressing at the same time as the topology. Do not renumber while you flatten; the anycast gateway and VRFs let you preserve addresses, and preserving them eliminates an entire class of application surprises.

Failure Domains and High Availability

Flattening the network changes what a failure means. In three-tier, a failed aggregation switch is a local event contained by STP and redundancy protocols. In spine-leaf, a failed spine reduces the fabric capacity by 1/N - with four spines, a 25% capacity loss, uniformly spread - and no path is ever fully lost as long as one spine survives. This is a much better failure profile than a blocking topology, but only if the ECMP set reselects fast.

That reselection is BFD's job. Tune BFD to a few hundred milliseconds at most, and make sure the underlay protocol is configured to remove the failed path immediately rather than waiting for a hello or hold timer. On the overlay, EVPN handles VTEP and MAC withdrawal; a leaf that loses its NVE peering to one spine simply stops using that spine's EVPN routes.

The one place a flattening design can genuinely hurt is the border. Where the fabric meets the WAN, the internet, or a legacy data centre, the border leafs carry all north-south traffic and most of the policy. Deploy at least two border leafs, keep the policy sets identical, and test a full border-leaf failure in the lab rather than discovering the behaviour during a maintenance window.

FAQ

Do I need VXLAN at all, or can I just run routing to the ToR? If no workload needs Layer 2 adjacency across racks and you do not need a mobile default gateway, a pure routed TOR design is simpler and has fewer failure modes. VXLAN and EVPN earn their complexity when VM mobility, multi-tenant segments or stretched VLANs are hard requirements.

What is the minimum number of spines? Two for production - four if you can afford it. Two spines with a leaf that has two uplinks means one spine failure costs half the fabric bandwidth, which is why most designs start at four spines with two uplinks per leaf, giving 2:N oversubscription of redundancy rather than 1:N.

How do I size the leaf uplinks? Multiply the number of server ports by their speed, divide by the oversubscription ratio you can justify, then choose the uplink combination that delivers it. Round up to a number divisible by the spine count so every uplink can be used in the ECMP bundle.

Does VXLAN need jumbo frames? It helps. Encapsulation adds 50 bytes (or 54 with options), so a 1500-byte inner frame becomes 1550 on the wire. Either raise the MTU to 1600/9216 across the underlay or clamp TCP MSS; mixing jumbo-capable and jumbo-incapable paths creates silent black holes for large packets.

Can I keep three-tier for the campus and use spine-leaf only in the data centre? Yes, and it is usually the right answer. Campus traffic is north-south and latency-tolerant; data centre traffic is east-west and latency-critical. There is no requirement to use the same architecture in both places.

More Data Center Design Content

Start with our leaf-spine two-tier design article and spine-leaf vs three-tier comparison, then choose hardware with the ToR switch selection guide. For the rest of the fabric build, see how STP blocking costs bandwidth, VXLAN EVPN multisite border gateway design, and VXLAN MTU and fragmentation with jumbo frames.

原文链接：https://www.layer23-switch.com/blog/spine-leaf-vs-traditional-architecture.html