Leaf-Spine Architecture: Two-Tier Data Center Design

2026年8月29日 LOG admin

The leaf-spine architecture has become the default data center network design for good reason: it collapses three tiers into two and replaces spanning-tree blocking with IP routing on every link. This article explains the design decisions behind leaf-spine — why three-tier networks stop scaling, how multichassis designs work, and what horizontal scaling really means — so you can defend your next architecture review with facts, not fashion.

The Traffic Pattern That Broke Three-Tier

Monolithic applications produced mostly north-south traffic: client to server and back. The classic three-tier design (access-distribution-core) handled that efficiently. Microservices, virtualization and AI flipped the balance to east-west traffic between servers, and every east-west flow in a three-tier network crosses the distribution and core twice — adding hops, latency and bottlenecks. The uplinks and core quickly become oversubscribed.

Quantify it once and the argument makes itself. A rack of 20 servers with dual 25G NICs can source 500 Gbps into a top-of-rack switch. If that ToR has two 40G uplinks to a distribution pair, its uplink capacity is 80 Gbps. The oversubscription ratio is roughly 6:1 on paper, and far worse in practice because two of those four 40G links may be STP-blocked in a traditional design. Storage replication, vMotion and distributed databases generate exactly this kind of sustained many-to-many load, so the fabric — not the server — becomes the limiting factor.

STP Wastes Bandwidth by Design

Layer 2 redundancy in a three-tier design relies on Spanning Tree Protocol, which blocks redundant links. Blocked links carry no traffic; as the network grows, STP blocks even more of it. Multichassis technologies (VSS, StackWise Virtual, MLAG) hide the redundancy pair as one logical switch so all links forward — but that still only fixes bandwidth waste, not scale. You are still pinned to a pair of chassis, and the pair is a fixed-capacity box you must eventually replace.

Two more failures follow from that design. First, convergence: a topology change triggers a spanning-tree reconvergence and a flood of unknown unicast across the whole L2 domain. A 4,000-VLAN bridging domain that hangs off one distribution pair is a large failure domain by definition. Second, path predictability: traffic takes whatever path STP permits, so latency between two racks depends on the current topology rather than on design intent. Neither property is acceptable for a latency-sensitive workload.

Leaf-Spine: Two Tiers, No STP, Predictable Paths

Leaf-spine introduces three changes at once:

Collapses access, distribution and core into leaf and spine only.
Runs IP routing on every link, removing STP entirely.
Every leaf connects to every spine, so any server-to-server path is exactly two hops — predictable latency and no blocked links.

Every link carries traffic; adding a spine or a leaf adds capacity in lockstep. Because the fabric is a uniform Clos topology, equal-cost multipath (ECMP) is available on every leaf: traffic leaving a leaf is hashed across all spines simultaneously, which is where the bandwidth comes from. The fabric has no blocked links, no root bridge and no topology-dependent latency.

Horizontal Scaling vs Vertical Scaling

Vertical scaling (scaling up) means replacing distribution/core switches with bigger chassis when demand grows — slow and expensive. Horizontal scaling (scaling out) adds more leaf switches and spreads load across devices, which is why hyperscalers and cloud providers standardize on it: flexible, highly available, and no hard ceiling. See our comparison of spine-leaf vs three-tier and the modern data center design overview for more depth, plus real-world considerations for 400G spine/100G leaf hybrid fabrics.

The operational difference is stark. Scaling up a chassis means a maintenance window, a cabling project and a serial-number-level hardware plan. Scaling out means adding one leaf, cabling it to the spine layer and letting the routing protocol converge — a change that touches a single rack and is naturally incremental in cost.

Bandwidth Math: Oversubscription and Spine Count

The two numbers that define a leaf-spine fabric are downlink capacity and uplink capacity. Take a leaf with 48 x 25G server ports and 8 x 100G uplinks:

Downlink (server-facing) bandwidth: 48 x 25G = 1,200 Gbps
Uplink (fabric-facing) bandwidth: 8 x 100G = 800 Gbps
Oversubscription: 1,200 / 800 = 1.5:1

Against four spines, that same leaf presents 8 x 100G, but the fabric-wide capacity depends on the spine count. With four spines each providing 8 x 100G toward the leaves, the fabric is symmetric: as long as every leaf uses all its uplinks, no single spine is oversubscribed relative to the others. Reduce the fabric to two spines and the same leaf can only bring up 2 x 100G of that capacity, degrading to 6:1 — an acceptable ratio for a general-purpose edge rack and a disaster for a storage or AI rack.

Three rules of thumb survive contact with real projects:

Keep the leaf-spine ratio constant. If you build 32 leaves against 4 spines, the next expansion to 64 leaves should also add spines, otherwise the spine layer becomes the bottleneck.
Pick an oversubscription target per rack class, not per fabric. Compute: 2:1 or better for storage and AI, 3:1 for dense virtualisation, 5:1–7:1 for general server racks.
Design for one spine failure. With N spines, a single failure leaves N-1 paths; if N is 2, that is 50 percent of fabric bandwidth gone in one event.

Reference Design: Address Plan, ASNs and Anycast Gateway

A leaf-spine fabric is a routing domain, so the design must specify addressing, autonomous systems and the default gateway model before any cable is patched.

Point-to-point links. Use /31 or /30 addresses on every leaf-spine link, taken from a dedicated block such as 10.0.0.0/16 carved per spine:

interface Ethernet1
   description LEAF1A-TO-SPINE1
   no switchport
   mtu 9214
   ip address 10.0.1.0/31
   no shutdown

BGP underlay. The standard RFC 7938-style model uses eBGP with one unique private ASN per leaf and a shared ASN for all spines. A shared spine ASN keeps the configuration identical across the spine layer and avoids path-hunting surprises. All leaf and spine loopbacks are advertised so any pair of devices has a stable, multipath-eligible route.

router bgp 65001
   router-id 10.255.0.11
   neighbor 10.0.1.1 remote-as 65000
   neighbor 10.0.1.1 description SPINE1
   maximum-paths 16
   address-family ipv4 unicast
      neighbor 10.0.1.1 activate
      network 10.255.0.11/32

Anycast gateway. For a VXLAN EVPN fabric, the default gateway must exist on every leaf simultaneously so a workload keeps its gateway after migration. That is the distributed anycast gateway, configured with the same virtual MAC and virtual IP on every leaf:

interface Vlan10
   no shutdown
   vrf member TENANT-A
   ip address 10.10.10.1/24
   fabric forwarding mode anycast-gateway

Keep the anycast MAC identical fabric-wide, keep the VLAN-to-VNI mapping uniform, and document both — a single leaf with a different virtual MAC produces a silent traffic blackhole for the hosts attached to it.

Physical Layer and Optics

The fabric is only as good as the cabling, and leaf-spine punishes sloppy physical design more than three-tier did, because every leaf needs a path to every spine.

Cabling topology. A full mesh from the leaf layer upward. Every leaf connects to every spine; there are no leaf-to-leaf links in the underlay.
Optics symmetry. Mixing 100G and 400G uplinks is workable but requires care. If a leaf has two 400G and four 100G uplinks, ECMP hashes flows across links of different capacity, and the 100G links become congested while the 400G links sit half idle. Either keep uplink speeds uniform per leaf or use weighted hashing if the platform supports it.
MPO trunk infrastructure. Structured cabling with MPO trunks between rows and breakout cassettes at the leaf is standard practice; it makes the “add a spine” operation a patching task rather than a cable-pulling project.
MTU. Set jumbo frames (typically 9214 bytes on Cisco, 9216 on Arista, 9216 on Juniper) consistently on every underlay interface, including loopbacks. A single interface left at 1500 silently fragments or drops VXLAN-encapsulated traffic.

Failure Domains and Convergence

Leaf-spine shortens the failure domain in two directions. Vertically, a leaf failure affects only the servers behind it; the rest of the fabric never notices. Horizontally, a spine failure removes one ECMP next hop and the remaining spines absorb the load within the BGP or IGP reconvergence window.

Useful targets and mechanisms:

BFD on every underlay link with sub-second timers, so a fibre or transceiver problem is detected in milliseconds rather than by BGP hold-timer expiry.
Maximum-paths equal to the spine count so ECMP actually installs all available paths.
Convergence budget: a well-tuned eBGP underlay with BFD reconverges in under a second. If you are measuring 30 seconds, the timers are the default and BFD is not running.
Routing-loop prevention. Every leaf advertises its own loopback as a /32; never redistribute server subnets into the underlay, or you build loops between the overlay and the underlay.

When Leaf-Spine Is the Wrong Answer

Leaf-spine wins when server-to-server traffic dominates, when the fabric must grow past the capacity of a single chassis pair, and when the same latency between any two hosts matters. It is a poor fit for a small campus with 200 users and one distribution pair, where the extra cables and the routing knowledge requirement buy nothing. It is also the wrong tool when the real requirement is segmentation by policy: a two-tier leaf-spine fabric does not replace firewalls, and micro-segmentation still needs a policy layer on top of the routing fabric.

FAQ

How many spines should a fabric have? Minimum two for redundancy, three or four for realistic capacity and failure headroom. Below two you have a single point of failure; above eight the cabling and optics cost grows faster than the benefit for most enterprises.

Do I have to use VXLAN EVPN? No. Leaf-spine can be a pure L3 fabric with plain routed links to the servers. VXLAN EVPN adds L2 extension for mobility and multi-tenancy; adopt it when workloads need subnet mobility or when you need many isolated tenants.

Can I mix leaf-spine with a legacy three-tier core? Yes, and it is the usual migration path. Terminate the old core on a pair of border leaves that also peer with the spines, keep the legacy L2 domains behind the core, and migrate rack by rack. See the spine-leaf vs traditional three-tier design and migration guide for the sequencing.

Is a two-tier leaf-spine design enough for a 10,000-server environment? Yes, provided the spine layer scales. The constraint is the number of fabric ports, not the tier count; a large spine layer with 64 or 128 ports per device supports very large leaf counts without adding a third tier.

How do I stop STP from creeping back in? Never extend a VLAN between leaves unless you are running EVPN for it, and do not create the classic “management VLAN across the whole fabric” L2 domain. Out-of-band management is the clean answer.