Spine-Leaf vs Three-Tier: Data Center Network Design

Modern cloud and virtualization workloads — VM mobility, containerized microservices, big data replication — have made east-west traffic the dominant flow in data centers, and the classic three-tier design is struggling to keep up. Spine-leaf (Clos) architectures flatten the network to two layers, remove Spanning Tree from the equation, and give every leaf the same number of hops to every other leaf. Based on Lenovo Press's introduction to spine-leaf networking designs, this article compares the two approaches and lays out the design rules and oversubscription math you need to size a fabric correctly.

This expanded guide goes further: it quantifies the oversubscription arithmetic with a worked example, explains how Layer 2 and Layer 3 fabric implementations differ, covers buffer behaviour and microbursts, and finishes with the hardware decision, the brownfield migration path and the verification commands that prove the fabric is healthy.

Three-Tier Architecture: Familiar but Limiting

The traditional design has core (L3), aggregation (L2/L3), and access (L2) layers. Its strengths are well known: pod-level fault isolation, security segmentation, and easy scale-out by adding pods. Its weakness is that pod-to-pod (east-west) traffic must traverse multiple layers, adding latency, and the core becomes an expensive, high-speed bottleneck as virtualization drives more east-west traffic.

The hierarchy was designed for a world where traffic was overwhelmingly north-south: clients reached servers, and the aggregation layer existed to concentrate access switches and provide default gateways. In that model a 20:1 oversubscription at the aggregation layer is invisible, because workloads rarely sustain more than a fraction of their access port. Virtualization invalidated that assumption. Live migration keeps memory pages synchronized across hosts, distributed storage replicates every write to two or three nodes, and container schedulers move east-west flows between racks continuously. East-west traffic can now be several times the north-south volume, and every one of those flows climbs the hierarchy.

The concrete limitations that show up in production are worth naming:

• Spanning Tree idles capacity. An access switch with two uplinks in an L2 topology leaves one link in blocking state, so half of the money you spent on uplinks buys no usable bandwidth.
• Oversubscription is concentrated at one layer. The aggregation layer is where thousands of access ports funnel into a small number of uplinks; it is simultaneously the most expensive and the most congested tier.
• Hop count varies with placement. Two hosts in the same pod may be two hops apart while two hosts in different pods are five hops apart, so application latency depends on wherever the scheduler placed the workloads.
• VLAN and failure domains grow with the pod. Large L2 domains mean large STP topologies, slower convergence, and broadcast traffic that consumes bandwidth on every link in the domain.
• Scaling means buying a bigger core. Adding a pod requires core ports, and core chassis scale in discrete, expensive steps rather than by adding inexpensive fixed switches.

Before deciding to flatten anything, measure the fabric you have. These commands show where traffic and unused capacity actually sit:

show spanning-tree summary totals
show interfaces counters rates
show etherchannel summary
show ip route summary

If the counters show that blocked uplinks would roughly double the available bandwidth, and that aggregation-to-core links run hot while access ports idle, the case for a flattened fabric is already made with data rather than with a diagram.

Spine-Leaf (Clos) Architecture

In a spine-leaf design, every leaf switch connects to every spine switch in a full mesh. The advantages:

• No STP loops — all interconnections are used simultaneously.
• Deterministic latency — east-west traffic is always equidistant (one hop through a spine).
• Fixed configurations — the switch config doesn't change as servers are added or moved.
• Scale-out — add spines for bandwidth, add leaves for server ports, without re-architecting.

The mesh can be implemented at Layer 2 (with TRILL or SPB) or Layer 3 (OSPF or BGP with ECMP). Layer 3 fabrics are increasingly preferred because routed ports now cost no more than switched ports and ECMP load-balances across all spines.

The name "Clos" comes from Charles Clos, who in 1953 described how to build one large non-blocking switch out of many smaller switches arranged in stages. A two-stage Clos — one leaf stage, one spine stage — is non-blocking as long as the aggregate bandwidth of the spine stage at least matches the aggregate access bandwidth of the leaf stage. That single inequality is the entire design; everything else is arithmetic about ports and oversubscription.

Two numbers govern the sizing. The first is switch radix, the total number of usable ports on the box. The second is the number of fabric-facing ports each leaf can spare. A leaf with 48 access ports and 6 uplinks of the same speed gives 8:1 oversubscription; with 8 uplinks it gives 6:1. Multiply leaves by uplinks to get the total spine-facing port demand, then divide by the spine's port count to learn how many spines you need. If that division does not come out even, the fabric is unbalanced and you should either change the uplink count or accept idle spine ports.

Routing across that mesh needs a scheme. Three patterns cover almost every deployment:

• eBGP with a unique ASN per leaf — the modern default in cloud-scale fabrics. Every leaf is its own autonomous system, spines sit in a shared transit ASN or peer directly, and ECMP is automatic because every spine is an equal-cost next hop for every leaf prefix.
• iBGP with route reflectors — common where the operations team already thinks in iBGP. Spines become reflectors, leaves are clients, and the same ECMP behaviour follows provided the IGP metric and next-hop handling are correct.
• OSPF or IS-IS underlay — simplest to run and fine for small and medium fabrics, but it scales less comfortably than BGP and is harder to steer with policy.

A minimal routed-fabric configuration on a leaf looks like this in EOS-style syntax:

router bgp 65101
   router-id 10.0.1.1
   maximum-paths 16
   neighbor SPINES peer group
   neighbor SPINES remote-as 65100
   neighbor 10.0.0.0 peer group SPINES
   neighbor 10.0.0.2 peer group SPINES
   neighbor 10.0.0.4 peer group SPINES
   redistribute connected

ECMP is only as good as the hash. Flows that produce the same five-tuple hash to the same spine, so one large flow cannot exceed a single uplink's capacity no matter how many uplinks exist. Where workloads generate a small number of very large flows, enable flowlets or use a hash with more entropy, then verify by looking at per-link rates rather than trusting the theory. Vendor-specific examples and hash policies are collected in our BGP ECMP configuration guide.

Layer 2 vs Layer 3 Fabric Implementation

Three implementation styles cover essentially every spine-leaf fabric in production:

• Routed fabric with no overlay. Every link is a /30 or /31, leaf ports are routed ports or SVIs, and each rack owns its own subnets. Fewest moving parts, no encapsulation overhead, but the addressing plan has to follow the physical topology, which makes cross-rack VM mobility harder.
• VXLAN with an EVPN control plane. A routed IP underlay carries a VXLAN overlay, EVPN distributes MAC and IP reachability, and any subnet can live on any rack. This is the dominant design for virtualization and container platforms. The cost is more state, more CLI, and an underlay MTU that must accommodate the 50-byte VXLAN header everywhere.
• Legacy L2 interconnect fabrics (TRILL, SPB). These solved the STP blocking problem before overlays existed but are now rare, mostly surviving in aging designs. New builds should not start here.

If you are weighing the STP argument specifically, the bandwidth a blocked three-tier uplink throws away is quantified in our article on STP blocking and usable fabric bandwidth. For an overlay that has to stretch subnets between data centers, see the VXLAN EVPN multisite border gateway design.

Overlay configuration on a leaf VTEP is a handful of lines:

interface Vxlan1
   vxlan source-interface Loopback1
   vxlan vlan 10 vni 10010
   vxlan vlan 20 vni 10020
   vxlan vrf TENANT-A vni 50001

Oversubscription Math and Sizing Rules

Key trade-offs to plan for:

• Each leaf must connect to every spine, so cabling and spine port counts grow quickly.
• Leaves typically have 4–6 spine uplinks; keep the leaf-to-spine oversubscription at 5:1 or better.
• For north-south traffic leaving the fabric, higher oversubscription (up to 100:1) may be acceptable.
• Three rules of thumb: every leaf needs the same number of fabric-facing ports; that count limits the number of spines; and the aggregate spine bandwidth times the oversubscription rate defines fabric edge bandwidth.

Worked example. Take a leaf with 48 x 25G access ports and 8 x 100G fabric uplinks. Access capacity is 48 x 25G = 1.2 Tbps and uplink capacity is 8 x 100G = 800 Gbps, so the leaf runs at 1.5:1 — effectively non-blocking for the server-facing ports and far better than the 5:1 rule of thumb. Split those eight uplinks as four to each of two spines so the leaf keeps an N+1 path set and ECMP spreads flows across both. Sixteen such leaves need 16 x 8 = 128 spine-facing 100G ports, or 64 ports on each of two spines; a 32-port 400G spine broken out to 4 x 100G per port supplies 128 ports per spine, leaving room to grow. Total fabric bandwidth is 16 x 800 Gbps = 12.8 Tbps, which the spine layer must be able to absorb for the fabric to stay non-blocking.

North-south sizing is a separate decision. Traffic leaving the fabric to a border router, a WAN link or object storage is usually bursty and tolerant of delay, so up to 100:1 oversubscription on egress links is common and acceptable. Storage replication, database mirroring and backup windows are the exceptions: those flows run at line rate for hours, so a 4:1 or better ratio on the links they cross keeps backups from eating the application traffic's headroom.

Check real oversubscription rather than the spreadsheet version:

show interfaces status
show interfaces Ethernet1-8 counters rates
show interfaces counters rates | include 100G

Latency, Buffers and Microbursts

Hop count is not the only latency term. Inside a switch, latency is the sum of the serialization delay on the ingress link, the forwarding delay through the ASIC, and the queueing delay when the egress port is contended. A three-tier path crosses three or more switching stages and queues at every oversubscribed boundary, so its latency varies with placement; a spine-leaf path crosses one or two stages with a fixed hop count, which is why design documents talk about deterministic latency rather than merely low latency.

Microbursts are the second-order effect that surprises people. A storage node with a 25G NIC can emit a full line-rate burst into a switch with a shallow buffer. If two or three such bursts arrive at ports sharing an egress, the switch must either buffer or drop, and fixed-form-factor switches built on merchant silicon have far less buffering than modular chassis. The practical mitigations in a flattened fabric are to keep access-to-uplink oversubscription modest, spread east-west flows across as many equal-cost paths as possible, enable ECN and PFC for lossless storage traffic instead of hoping buffers absorb it, and watch for elephant flows pinned to one hash bucket.

Choosing Hardware for Your Fabric

Fixed-form-factor switches dominate the leaf layer (e.g. 32x100G or 64x100G devices), while chassis devices with multiple line cards make high-density spines feasible. Evenly distribute leaf uplinks across spine line cards to reduce fabric-module traffic. For practical fabric examples, see our 400G spine / 100G leaf link consistency article, the VXLAN MP-BGP EVPN design guide, and Arista EOS MLAG explainer for multi-chassis leaf pairs.

Beyond form factor, three hardware properties drive the choice:

• Switching silicon and feature depth. The ASIC sets the ceiling on routes, ACL entries and overlay endpoints. A leaf acting as a VTEP needs overlay-aware silicon and enough tunnel endpoint capacity; a spine that only forwards IP needs little beyond a large FIB.
• Buffer size. Deep-buffer chassis matter where storage, WAN and incast traffic meet; shallow-buffer fixed switches are fine for evenly distributed east-west flows.
• Optics and cabling. A 400G spine with 100G leaf uplinks needs breakout cables, and the cable plant is often the least flexible part of the design. Choose optics that let you change uplink counts without changing physical infrastructure.

Also budget the mesh itself. Each spine you add costs one port per leaf, so a fabric with 40 leaves and 4 spines spends 160 ports on spine-facing links. Growth in the leaf tier forces growth in the spine tier, and the port cost of the mesh belongs in the initial design rather than in a surprise refresh two years later.

Migrating from Three-Tier to Spine-Leaf

Most real migrations are brownfield. A sequence that works reliably is: build the spine-leaf fabric alongside the existing three-tier network with its own addressing and routing; move storage and application clusters that talk to each other onto leaf pairs, keeping VLANs local to the leaf wherever possible; use MLAG (or EVPN multihoming) on leaf pairs so servers dual-attach without STP and without a first-hop single point of failure; interconnect the two networks only at the border; and retire the aggregation and core tiers once the last workload has moved and monitoring shows no traffic.

The classic mistake is stretching a VLAN across both networks. That re-introduces exactly the STP domain you are trying to escape and makes fault isolation impossible. A phased plan with per-rack cutover and a documented rollback beats a big-bang migration every time; the step-by-step approach is covered in our spine-leaf migration guide.

Verifying and Troubleshooting the Fabric

Once the fabric is live, these are the commands worth running after every change and on a weekly schedule:

show lldp neighbors
show ip route summary
show ip bgp summary
show ip route 10.20.30.40
show interfaces counters errors

Read them in this order. show lldp neighbors proves the physical topology matches the design and catches a cable in the wrong port. The routing and BGP summaries confirm that every leaf has exactly the expected number of equal-cost paths to every other rack prefix. A route lookup for a specific destination shows which path a flow will take, which is how a missing ECMP member is found before it becomes a support ticket. The counter commands then show whether traffic is actually spread across those paths or piling up on one.

When Three-Tier Still Makes Sense

Spine-leaf is not automatically correct everywhere. A three-tier or collapsed-core design remains reasonable where traffic is genuinely north-south, where the topology is small enough that a two-switch core provides all the port density needed, where the operations team has deep STP and first-hop-redundancy skills and no appetite to run BGP, or in a campus with distribution closets and long cable runs. The decision should follow the traffic matrix: if the measurement shows that most bytes stay inside a rack or leave the building, a full Clos fabric with an overlay buys complexity rather than performance.

Key Takeaways

• Spine-leaf exists because east-west traffic became dominant; the flat equal-cost topology is the response, not a fashion.
• Sizing starts with two divisions: leaves multiplied by uplinks gives spine-facing port demand, divided by spine ports gives the spine count.
• Keep leaf uplink oversubscription at 5:1 or better, and accept up to 100:1 only on north-south egress.
• Route the fabric with eBGP or iBGP plus ECMP, and verify that the hash spreads real flows rather than trusting theory.
• An overlay buys mobility and costs MTU, state and complexity — choose it deliberately.
• Migrate in phases, one rack at a time, and never stretch a VLAN between the old and the new fabric.

原文链接：https://lenovopress.lenovo.com/lp0573.pdf