Data Center ToR Switch Selection: Key Specs and Comparison Guide

Buying the right ToR (Top-of-Rack) switch decides the performance, reliability and upgrade path of your entire data center rack - yet most procurement decisions are made on port count alone. This guide walks through the specs that actually matter: port density and speeds, switching capacity vs real throughput, buffer depth for lossless workloads, latency, L2/L3 feature set (MLAG, VXLAN/EVPN), optics compatibility, management and total cost of ownership. A comparison of common 25G/100G ToR models is included as a starting point for your own shortlist.

The Specs That Matter

1. Port Density and Speeds

• 48 × 10/25G downlink ports is the current sweet spot for server connectivity.
• Uplinks: 6-8 × 100G (or 400G on newer platforms) toward the spine.
• Breakout support (100G → 4×25G, 40G → 4×10G) gives flexibility for mixed server generations.

The asymmetry is intentional. A rack with 20 servers at 2 × 25G is 40 downlink ports; eight 100G uplinks provide a 4:1 oversubscription ratio, which is the usual starting point for general compute and comfortably supports 2:1 for storage-heavy racks. Write the oversubscription ratio you are willing to accept on the purchase order, because it — not the port count — is what determines whether the rack survives a busy afternoon.

2. Switching Capacity and Forwarding Rate

Compare real throughput, not marketing numbers: a 48×25G + 8×100G switch should forward ~3.2-4 Tbps non-blocking. Check the forwarding rate (Mpps) matches the capacity at minimum frame size. A switch can advertise 4 Tbps of switching capacity and still be unable to forward 4 Tbps of 64-byte frames, because the bottleneck in that case is the packet-per-second rate rather than the bit rate. For every candidate, divide the stated Mpps by the stated Tbps and sanity-check it against line rate: it should be consistent with a 64-byte frame.

3. Buffer Depth

Storage (NVMe-oF, RoCE) and AI workloads are loss-sensitive. Larger shared buffers (16-32 MB) absorb micro-bursts; look for per-port QoS and PFC (Priority Flow Control) support for lossless Ethernet. Buffer architecture matters as much as buffer size: a dedicated per-port buffer pool with a shared overflow space behaves very differently under incast than a single fully shared pool, and a shallow shared pool on a ToR uplink is the classic cause of PFC storms that spread across the whole rack.

4. Latency

Cut-through vs store-and-forward: most data center switches do cut-through at sub-microsecond port-to-port latency. For HPC/AI fabrics, latency matters; check the vendor's published numbers. Where latency is not the primary objective, prefer cut-through anyway for the tail-latency benefit, but confirm that features you need (some ACL, VXLAN or flow-based telemetry paths) do not silently switch the port to store-and-forward.

5. Power, Cooling and Physical Fit

Per-port watts at typical and full load, airflow direction (front-to-back is standard; others exist for specific cabinet layouts), 1U vs 2U depth versus your rack, and PSU redundancy all belong in the comparison. A switch that draws 350 W more than an alternative costs real money across 40 racks and may not fit the power budget of the cabinet you already own.

6. L2/L3 Feature Set

• MLAG (multi-chassis LAG) for active-active ToR pairs - mandatory in production racks.
• VXLAN/EVPN for overlay fabrics - required for any modern leaf/spine design.
• BGP/OSPF, PIM, ACLs, sFlow/NetFlow, and a programmable API (e.g., EOS eAPI, NX-API, REST).

Test the features you depend on rather than the ones on the datasheet. The most common gaps are EVPN multi-homing type, VXLAN ACLs on the overlay, and whether MLAG can be upgraded without a full rack outage.

Comparison of Common ToR Switches

Read the table as five different philosophies rather than five interchangeable boxes. On NX-OS, MC-LAG capabilities vary by platform and release and should be agreed with the vendor's account team; a switch that can only do the basics will need a second control plane or a redesign.

25G, 100G or 400G to the Server?

25G per server uplink remains the default because it matches the realistic bandwidth of a dual-socket general-purpose server and keeps the per-rack optics cost low. 100G-to-server is now justified for NVMe-oF storage nodes, GPU training nodes and any host that can actually push more than 50 Gbps of useful traffic. The practical question is not "what is fastest" but "how many racks will need 400G uplinks within the depreciation period of this switch". If the answer is more than a handful, buy the platform with 400G-capable uplink cages now: the ports are usually the cheapest part of the upgrade.

Optics and Cabling

• Match optics to reach: DAC (1-5 m intra-rack), AOC (5-30 m), SR4/SR8 multimode (100 m), LR4/LR8 single-mode (2-10 km).
• Check the vendor's supported transceiver list - using unsupported optics may void support and lock features.
• Consider 400G-ready uplinks even if you start at 100G - the optics can be cheaper than a full switch swap later.

Budget the optics explicitly and per rack: on a 48-port ToR, the transceivers frequently cost more than the switch. Cable management is a design input too - MPO trunk plant between racks is far easier to certify and re-patch than a bundle of individual AOCs, and it gives you a migration path from 100G-SR4 to 400G-SR8 on the same fibre.

Redundancy Models: MLAG vs EVPN Multi-Homing

Every production rack pairs two ToRs, and the choice of how to pair them affects both the failure domains and the operations model.

Verify what interlocks the platform has before you commit: some MLAG implementations block any control-plane upgrade that touches the peer, while EVPN multi-homing upgrades each member independently. For the failure domains and bandwidth implications of these designs, see Spine-leaf vs three-tier: STP blocking and bandwidth and Leaf-spine vs traditional three-tier topology.

Management and Automation

Evaluate the management story: CLI quality, REST/gRPC APIs, telemetry (streaming), integration with Ansible/Terraform, and the vendor's automation platform (CloudVision, NDFC, NetEdit). A switch that is painful to automate will hurt for years. The three questions that separate platforms fastest are:

• Can I render the entire per-rack configuration from a template and apply it idempotently?
• Can I get per-queue and per-ECN counters by streaming telemetry, without polling SNMP?
• Can the device be zero-touch provisioned and then replaced by the same template with no manual steps?

If you plan to standardise on one vendor across the fleet, the automation ecosystem is often more valuable than a 10% price difference. For the underlay itself, the classic L2 VLAN/trunk design still matters in the first racks: see ArubaOS-CX access vs trunk verification for how to prove a trunk is actually passing what you think.

Proving the Platform Before You Buy

Ask for a loaner and run these checks rather than trusting a demo:

show version
show interfaces status | include notconnect
show interfaces counters errors
show interfaces counters queue | no-more
show platform resources
show system internal buffer-pool   # vendor keywords vary
show vxlan address-table
show evpn mac vni 10000

Specifically look for: error counters that move under load (physical or ASIC-level), queue counters that reveal a persistently congested uplink queue, PFC pause counters when you exercise a lossless class, and the time a full EVPN configuration load takes on the control plane. Then run one failure test — pull an uplink and a ToR PSU — and measure the convergence time you actually get.

Validation Checklist for the POC

• Line-rate forwarding at 64-byte frames on every port simultaneously.
• Lossless class behaviour with PFC and ECN under incast; check pause frames are the exception, not the steady state.
• MLAG or EVPN multi-homing failover, with measured traffic loss, not just a log entry.
• VXLAN/EVPN bring-up, including VM mobility and MAC learning across racks, and the MTU consequences of the overlay.
• Software upgrade and rollback with a per-rack maintenance window that fits your change policy.
• Ansible/Terraform apply twice in a row, to prove the configuration is idempotent.

TCO Checklist

• List price vs street price; factor optics, stack cables, licenses.
• Power and cooling: per-port watts at full load, airflow direction vs your rack layout.
• Warranty and support: NBD vs 4-hour; software subscription costs.
• Fleet consistency: standardize one ToR model per rack type to simplify spare parts and automation.

Add two lines that procurement spreadsheets usually miss. First, the cost of spares across the depreciation period: three models per site means three separate spare pools plus three firmware baselines. Second, the cost of the migration you will eventually do anyway — the discount you negotiate today is worth little if the next generation of the platform cannot use the same optics or the same automation templates.

Common Procurement Mistakes

• Buying on port count alone. A 48-port switch with a shallow shared buffer and a 25 Mpps shortfall will drop traffic long before it runs out of ports.
• Ignoring the uplink oversubscription. Eight 100G uplinks behind 48 × 25G is 3:1, not 1:1. That is fine for compute and wrong for storage.
• Assuming breakouts are free. Splitting a 400G port into 4×100G may consume a licence, and each breakout optic has its own supported-transceiver constraints.
• Mixing models per rack. Two ToR models in one fabric means two firmware baselines, two spare pools and two sets of automation quirks.
• Skipping the automation test. A platform with no usable API or with non-idempotent config pushes costs far more over five years than the purchase price difference.
• Forgetting optics and cables in the TCO. On a 48-port ToR the transceivers frequently exceed the switch cost; model them per rack before you sign.

Decision Flow

• If the rack is all general-purpose compute → 48×25G with 8×100G uplinks and MLAG.
• If the rack hosts NVMe-oF or GPU nodes → 100G-to-server, deep shared buffers, PFC/ECN, and EVPN for the fabric.
• If the fabric is multi-vendor or must scale past a handful of racks → prefer the standards-based EVPN multi-homing model over proprietary MLAG.
• If the rack must survive for a hardware refresh cycle → confirm the uplink cages can run 400G optics on the existing plant.

Related: Leaf-spine two-tier design and 400G spine / 100G leaf link consistency. For the lossless-Ethernet configuration that the storage racks depend on, see RoCEv2 lossless Ethernet: PFC and ECN, and for AI clusters specifically InfiniBand vs RoCE vs Ethernet.

原文链接：https://www.arista.com/en/solutions/