Leaf-Spine vs Three-Tier: Oversubscription and ECMP Design

The argument between a two-tier leaf-spine fabric and a classic core/aggregation/access hierarchy is not about fashion, it is about traffic direction and how much of the installed bandwidth you are allowed to use. This article sets out the numbers that decide the question: oversubscription ratios, blocked links under spanning tree, and the cost of scale-out versus scale-up.

What changed: north-south became east-west

Three-tier networks were designed when clients talked to servers. Access, aggregation and core each added a hop and a redundant pair designed for north-south flows. Virtualisation, microservices and distributed storage flipped the pattern: most traffic is now server-to-server inside the data centre, and it happens to cross the hierarchy twice — up to the distribution and core, then back down.

Bandwidth you cannot use

The access layer connects to distribution at Layer 2 and therefore relies on spanning tree. STP is a loop-prevention protocol, not a load-sharing protocol: on a redundant pair of uplinks it blocks one and leaves it idle. Multichassis techniques (VSS, StackWise Virtual, vPC, MC-LAG) hide the redundancy from STP so both links forward, which fixes the waste but not the topology.

Oversubscription: the numbers that matter

• Leaf to spine: with four to six spine uplinks per leaf, a ratio around 5:1 is generally considered acceptable for production east-west traffic.
• Spine to core: because the fabric is optimised for east-west, much higher ratios (commonly quoted up to 100:1) are tolerable for north-south egress.
• Three-tier access to distribution: whatever the design says, only the forwarding links count — the blocked ones contribute zero.

Every leaf connects to every spine in a full mesh, so inter-leaf traffic always crosses exactly two hops. That predictability is worth as much as the raw bandwidth in latency-sensitive workloads.

Scale-out versus scale-up

# Layer 3 leaf-spine underlay (per leaf, Cisco-style)
router bgp 65001
 neighbor 10.0.1.0 remote-as 65001
 neighbor 10.0.1.0 update-source Loopback0
 address-family ipv4 unicast
  maximum-paths 4

Adding capacity in a spine-leaf fabric means adding a leaf for server ports or a spine for inter-switch bandwidth. In three-tier it means replacing the distribution or core chassis — a slow, expensive, outage-prone exercise. BGP with ECMP plus loopback-based peering is the standard underlay precisely because it scales horizontally with no re-architecture.

The trade-offs people forget

Full-mesh cabling is real work: every leaf needs a link to every spine, and port counts on the spine cap how many leaves you can attach. Load balancers, firewalls and storage arrays may still prefer the pod-based isolation a three-tier design gives for free. Choose the fabric where east-west traffic dominates, and keep a hierarchical pod design where regulation, tenancy or legacy appliances demand strict separation.

Related reading: StackWise Virtual dual-active detection, Microbursts and switch buffer sizing, Loopback interface design.

原文链接：Introduction to Spine-Leaf Networking Designs (Lenovo Press)