InfiniBand vs RoCE vs Ethernet: Choosing an AI Cluster Fabric

The back-end fabric decides whether GPUs spend their time computing or waiting on all-reduce. Three families are in play today: InfiniBand, RoCEv2 over Ethernet, and vendor-scheduled Ethernet such as Spectrum-X-style fabrics with telemetry-driven congestion control. This guide compares them on the axes that actually drive a purchase decision - effective throughput for collectives, operational burden, cost per port and lock-in - and gives a decision rule you can defend in a design review rather than link-flapping between vendor benchmarks.

The short answer

For large dedicated training pods bought as a single NVIDIA system, InfiniBand (or NVLink for the intra-node domain) is still the default: lossless by construction, credit-based flow control in hardware, SHARP in-network reduction, and one vendor for fabric, NICs, optics and diagnostics. For inference fleets, multi-vendor strategies, and anyone whose NetOps team already speaks Ethernet, lossless RoCEv2 or a scheduled Ethernet fabric is now credible at very large scale.

Where each option wins

What really determines GPU-fabric readiness

Three checks decide whether a fabric can carry training traffic, regardless of protocol:

! 1. Non-blocking bandwidth where it is needed - oversubscription math
leaf_downlinks_x_speed / leaf_uplinks_x_speed = oversubscription ratio

! 2. Lossless verification - are pause and ECN counters quiet under load?
show interface priority-flow-control
ethtool -S ens1f0 | grep -Ei 'pause|pfc|discard'

! 3. End-to-end MTU consistency - RDMA breaks on a single 1500-byte hop
ping -M do -s 8972           # Linux
ping 10.0.0.2 size 9000 df-bit     # NX-OS / IOS

A one-tier oversubscribed fabric will not be fixed by a better protocol. Likewise, a single 1500-byte hop in the path silently caps MTU and produces bizarre, hard-to-trace performance regressions.

Fabric management is the hidden cost

InfiniBand requires a subnet manager (embedded in switches or UFM) to discover the topology, assign LIDs and program forwarding. Ethernet needs none of that, but it does need PFC/ECN hygiene, consistent QoS markings and - in scheduled fabrics - a telemetry/controller stack. Count the headcount, not just the bills of materials: a fabric nobody on the team can debug is the most expensive option on the list.

Topology notes for GPU clusters

Rail-optimised designs keep every GPU of the same rail index on one leaf, so collective traffic stays within one hop for the common case and only the final all-reduce crosses the spine. This reduces hot spots far more effectively than adding tiers. At 400G and beyond, copper DAC reach collapses quickly - plan for active copper cables or optics and budget their power and cooling, which for a 200kW+ training pod is not a rounding error.

Decision rule

Choose InfiniBand when training throughput is the product and you are buying a single-vendor system. Choose RoCEv2 or scheduled Ethernet when you must amortise one operational model, want multi-vendor pricing, or the workload is inference-heavy. In both cases, validate before sign-off with real collective microbenchmarks - nccl-tests all-reduce at several message sizes plus a bandwidth sweep with ib_write_bw or perftest - rather than trusting a datasheet. Related guides: RoCEv2 lossless Ethernet with PFC and ECN, spine-leaf architecture design, ToR switch selection and automating fabric design with Ansible.

原文链接：https://ai-infrastructure.net/networking-fabric