NVIDIA MLNX-OS LAG and LACP Configuration Guide

NVIDIA Mellanox switches running MLNX-OS are common in HPC and storage fabrics, and link aggregation is usually the first feature you configure: bonding two or four 25/40/100G ports into one logical port-channel doubles the bandwidth and removes single-link risk. This guide walks through MLNX-OS LAG and LACP configuration from scratch: creating the port-channel, assigning member ports, choosing static vs LACP modes, and verifying the bundle with the right show commands.

Step 1: Create the Port-Channel

switch (config) # interface port-channel 1
switch (config interface port-channel 1) # switchport mode trunk
switch (config interface port-channel 1) # switchport trunk allowed vlan add 10,20,30
switch (config interface port-channel 1) # exit

MLNX-OS uses interface port-channel <id> like IOS. The port-channel inherits switchport settings; here it is a trunk carrying VLANs 10, 20 and 30.

Step 2: Assign Member Ports with LACP

switch (config) # interface ethernet 1/1
switch (config interface ethernet 1/1) # channel-group 1 mode active
switch (config interface ethernet 1/1) # exit
switch (config) # interface ethernet 1/2
switch (config interface ethernet 1/2) # channel-group 1 mode active
switch (config interface ethernet 1/2) # exit

mode active enables active LACP (802.3ad) - the switch actively sends LACPDUs. Use mode passive to wait for the peer, or mode on for a static (non-LACP) bundle when the peer does not support LACP.

Step 3: Save the Configuration

switch (config) # exit
switch # write memory
Success.

Static vs LACP: Which Mode Should You Use?

MLNX-OS gives you three ways to build a bundle, and the choice has real operational consequences:

• mode active: the local switch sends LACPDUs and negotiates with the peer. Both sides must agree before a member joins the bundle. This is the recommended mode on both ends of a link.
• mode passive: the local switch waits for the peer to initiate. Use it only when the peer is fixed at active; two passive ends will never form a bundle, and one active plus one passive is the normal interop pattern.
• mode on: a static bundle with no LACP at all. The ports come up as soon as they are physically up. It is tempting because it "just works", but it is dangerous: if a link is mis-wired to the wrong switch or a member is duplicated, the static bundle will happily forward into a loop. Use it only where the peer genuinely has no LACP support.

The rule of thumb is simple — if both devices can speak LACP, run active/active. LACP catches cabling mistakes, detects dead links faster than carrier loss alone, and prevents the half-configured bundle that silently black-holes traffic.

Verification Commands

show interfaces port-channel 1
show interfaces port-channel summary
show lacp
show lacp interfaces
show interfaces ethernet 1/1
show interfaces counter port-channel 1

show interfaces port-channel 1 shows the bundle state and member ports; show lacp interfaces displays LACP partner state for each member (look for BNDL/Collecting-Distributing state - if members show AGG but not BNDL, the peer is not matching the LACP config). show interfaces counter port-channel 1 is the fastest way to confirm traffic is actually flowing across the aggregate rather than pinned to one member.

Common Issues and Fixes

Port-Channel Load Balancing

switch (config) # port-channel load-balance ethernet dst-ip
switch (config) # port-channel load-balance ethernet l4-src-port

MLNX-OS lets you pick the hash fields for the bundle; for storage/NFS traffic, include L4 ports to spread flows across members. The hash is computed per flow, not per packet, so every packet in a given conversation follows the same member — which preserves ordering but also means an unlucky hash can pin a heavy flow to a single link. Choose the fields that match your dominant traffic: source and destination IP for routed traffic, and L4 source and destination ports when many flows share just a few IP pairs.

Sizing a LAG: How Many Members Is Right?

More members are not always better. Link aggregation gives you active/active throughput and resilience, but the achievable aggregate is bounded by the slowest common denominator and by how well your flows spread. A few practical rules:

• Keep all members at the same speed and duplex. Mixing 25G and 100G members in one bundle creates uneven distribution and unpredictable failover.
• Two members are usually enough for redundancy. Three or four help only when flows are diverse; a single elephant flow will still ride one member.
• Spreading a LAG across two line cards, or two switches, buys you resilience against a card or chassis failure. A four-port bundle that shares one ASIC loses everything if that ASIC fails.
• For storage traffic, size the LAG for the worst case of one member failing. If the array needs 80G and you have two 40G members, losing one drops you to 40G.

Interoperating with Servers and Non-Mellanox Switches

Bundles almost always terminate on something that is not the same vendor. A few interop notes save hours of head-scratching: Linux bonds should be configured as mode 4 (802.3ad) with the same hash policy and an LACP rate matching the switch; a Dell or Arista peer should use the equivalent channel-group mode; and the MTU must match across every member and the port-channel interface. If the peer expects jumbo frames and the bundle does not, LACP will come up and then mysteriously pass almost no traffic.

Monitoring and Capacity Planning

A working bundle is not the same as a balanced one. Watch per-member counters over a full business cycle and compare them: if one member carries the bulk of the traffic, the hash is a poor fit or the flows are too few. Trend the port-channel utilisation so you learn when to add members before saturation bites, and keep an eye on LACP state transitions — a member that keeps bouncing in and out of the bundle usually points at a marginal optic, a failing cable or a speed mismatch rather than a software problem.

Cross-Switch Bundles: LAG vs MLAG

A plain port-channel bonds ports on a single switch. If that switch fails, the bundle dies with it, and a server or storage array dual-homed to two switches needs something stronger: a multi-chassis link aggregation group. MLNX-OS supports this as MLAG, where two switches present one logical LAG to the peer. The configuration looks similar but adds a peer-link between the switches and an IPL (inter-peer link) carrying the control traffic:

switch-1 (config) # mlag-vip my-mlag-vip ip 10.10.30.1 /24 force
switch-1 (config) # interface port-channel 10
switch-1 (config interface port-channel 10) # exit
switch-1 (config) # mlag 10
switch-1 (config mlag 10) # exit
switch-1 (config) # interface ethernet 1/1
switch-1 (config interface ethernet 1/1) # channel-group 10 mode active
switch-1 (config) # interface port-channel 100
switch-1 (config interface port-channel 100) # exit

The important distinction is where the hash and failover decisions are made. With a single-switch LAG, all members live under one control plane, so balancing and failover are local. With MLAG, the two switches coordinate through the IPL and must agree on which member is forwarding; a broken IPL or a version mismatch between the peers can leave both sides believing they own the bundle. Verify MLAG state with show mlag and show mlag-vip, and treat the peer-link as a critical path — losing it is a bigger event than losing one member.

Reading the LACP Flags Like a Troubleshooter

When a bundle misbehaves, the LACP state machine is your best diagnostic. show lacp interfaces lists a set of flags per member; the ones that matter are A (activity), B (bundled), C (collecting), D (distributing) and S (synchronised). A healthy member shows bundling, collecting and distributing all set. The failures fall into a few patterns:

• AGG without BNDL: the member is aggregating physically but has not been bundled by LACP, which almost always means a mode, key or partner-identifier mismatch.
• Synchronised but not collecting/distributing: usually a transient state during negotiation, or a speed/duplex mismatch between the two ends.
• Member flaps between states: a marginal optic, a bad cable or the wrong SFP type. Swap the transceiver before you change the config.

Because LACP keys the bundle on the partner's system ID and port state, an asymmetric configuration — one side trunking VLAN 10 and the other side trunking VLAN 20 — will still bundle but will not pass the traffic you expect. That is why "identical member configuration" is not a style preference but a functional requirement.

Step-by-Step Bring-Up Order

Order matters when you bring a bundle into production. Work from the interface inward, and verify at each step rather than at the end:

switch (config) # interface ethernet 1/1
switch (config interface ethernet 1/1) # mtu 9216
switch (config interface ethernet 1/1) # exit
switch (config) # interface port-channel 1
switch (config interface port-channel 1) # mtu 9216
switch (config interface port-channel 1) # exit
switch # show interfaces port-channel summary

Set the MTU on both the physical members and the port-channel interface — a mismatch between the two is a classic cause of "the bundle is up but nothing works". Then bring the members up one at a time and confirm each joins the bundle before moving to the next, so a problem is isolated to a single port rather than buried in a four-port tangle.

Best Practices

• Use LACP (mode active) instead of static bundles where possible - it detects mis-wiring and dead links.
• Keep all members on the same speed and VLAN settings.
• For storage fabrics, verify the hash policy spreads traffic or a single member will saturate.
• Save the configuration and back it up off the switch before you leave.

Related: Mellanox switch port types in MLNX-OS, Dell OS10 port-channel and LACP configuration, the MLNX-OS first-time setup wizard walkthrough, and Linux bonding 802.3ad LACP rate and hash policy.

原文链接：https://docs.nvidia.com/networking/display/MLNXOSv31120/