MLNX-OS: Troubleshooting Breakout Cable Link Issues

Breakout cables are the cheapest way to multiply 100G/200G ports, but a link that refuses to come up on a split port can waste hours if you start swapping hardware. NVIDIA's support article on breakout-cable link issues lays out a disciplined isolation order, and this guide follows it: reset the port-level split configuration so auto-sensing can work, verify the split actually took, then methodically rule out firmware, cable part number, MFT tooling and LID mistakes before touching the physical layer. Most 'bad ports' turn out to be configuration or tooling issues.

A. Revert Port-Level Configuration

Reset split configuration to default so the switch auto-detects and applies the correct port split:

mlxconfig -d -lid-<LID> set SPLIT_PORT[1..64]=0

B. Query Port Split Status

Verify the split was applied and look for the expected mode, e.g. SPLIT_2X(1):

mlxconfig -d -lid-<LID> q | grep SPLIT_PORT

C. Check Server Adapter and Firmware

Confirm the server adapter firmware matches firmware of servers with known-good split configurations:

ibv_devinfo

Firmware inconsistency is a common cause of split-port negotiation failures.

D. Confirm Breakout Cable Model

The breakout cable must be the same part number as cables proven in other setups. Mismatched or unsupported cables prevent correct link negotiation even when the port config is right.

E. Isolate the Problem

Move the cable from the failing port (Port X) to a known-good split port (Port Y):

• Link works on Port Y → the problem is Port X (hardware or config on that port).
• Link still fails → the problem is the cable or the server NIC.

F. Validate MFT Version

Check that Mellanox Firmware Tools (MFT) are compatible with the switch firmware — mismatched versions cause failed configs and invalid updates:

mst status
flint -i <firmware>.bin --version

G. Confirm LID Number

A wrong Local Identifier (LID) silently applies configuration to the wrong device. Verify the switch LID:

ibnetdiscover | grep -i "Switch"

Worked Example

# 1. Reset split config
mlxconfig -d -lid-10 set SPLIT_PORT[1..64]=0
# 2. Check split status — expect SPLIT_2X(1)
mlxconfig -d -lid-10 q | grep SPLIT_PORT
# 3. Verify firmware
ibv_devinfo
# 4. Match cable P/N against working breakout cables
# 5. Move cable to Port Y and observe link
# 6. Validate MFT version vs switch firmware
# 7. Use the correct LID everywhere

Related: NVIDIA MLNX-OS configuration management and the ONIE/Onyx MLNX-OS installation guide.

原文链接：https://enterprise-support.nvidia.com/s/article/Troubleshooting-Link-Issues-with-Breakout-Cables-on-MLNX-OS-Switch-Ports