Mellanox InfiniBand Switch Setup: Enable Subnet Manager

2026年8月25日 LOG admin

An InfiniBand fabric will not come up until a subnet manager is running - the lights stay amber and the links never start. This practical tutorial from Lambda Labs walks through enabling the subnet manager on a Mellanox SB7800 36-port EDR switch, both from the CLI and through the MLNX-OS web console. It covers discovering the switch's IP address with arp-scan, SSH login with the default credentials, enabling the subnet manager with a single command, and verifying that the fabric is operational.

What Is the InfiniBand Subnet Manager?

The subnet manager discovers and configures the devices running on the InfiniBand fabric. It assigns local identifiers (LIDs), establishes path tables and manages the fabric topology. Without an enabled subnet manager, the switch cannot activate its ports - a common first-time setup mistake.

What the Subnet Manager Actually Does

It helps to understand the subnet manager as the control plane of an InfiniBand fabric. Unlike Ethernet, where forwarding tables are learned hop by hop from data-plane traffic, InfiniBand is a centrally managed network: nothing forwards packets until an entity has walked the topology and programmed every switch and every channel adapter.

A subnet manager performs these jobs in sequence during what is called a sweep:

Discovery. It reads the subnet management attributes from every node and port, building a map of the physical topology — which HCA is connected to which switch port, at what width and speed.
LID assignment. Every port receives a 16-bit Local Identifier. LIDs are the addresses used for routing inside a single subnet, and each node learns the LID of every other node from the SM.
Path-table programming. The SM computes routes and writes Linear Forwarding Tables (LFTs) into every switch, so that unicast traffic follows a deterministic path with no per-hop decisions.
Multicast forwarding tables. Multicast groups (used heavily by MPI and by RDMA collectives) get their forwarding entries installed on the switches.
Service Level and QoS configuration. Virtual lanes, SL-to-VL mapping and the parameters that support congestion control are pushed to the fabric.
Ongoing maintenance. After the initial sweep the SM keeps running, re-scanning at a configurable interval to detect new links, dropped links and topology changes, and reprogramming as needed.

Subnet Manager Roles: Master, Standby and Priority

You can — and in production you should — run more than one subnet manager on the same fabric. Each SM instance has a priority value; the instance with the highest priority becomes the master and does the real work, while the others sit as standby instances that periodically poll the master. If the master stops responding, a standby takes over within seconds and re-sweeps the fabric. On a Mellanox switch the embedded subnet manager is configured per SM node, which is why the CLI command references smnode. Setting a lower priority on the switch SM lets you run OpenSM on a compute node as the intended master and keep the switch as a backup, or vice versa.

Embedded Subnet Manager vs OpenSM on a Host

There are two common ways to run a subnet manager, and it is worth choosing deliberately:

Embedded SM on the switch (MLNX-OS). Zero extra infrastructure: the switch manages the fabric itself, it is always powered on when the fabric is, and there is no dependency on a server that might be rebooting. Managing it is a handful of CLI commands. This is the simplest correct answer for small and medium fabrics.
OpenSM on a compute node. Runs as a service on a host (the opensm package on Linux). It offers more knobs — routing engines, QoS policy files, per-fabric tuning — and is the traditional choice in large HPC fabrics where the SM is centrally managed. Its weakness is that the fabric depends on a server staying up; if that node reboots, the fabric re-sweeps when it returns.

For a lab or a single-switch cluster, enable the embedded SM and be done. For a large multi-switch fabric, run OpenSM on a dedicated node (or two) and use the switch SM only as a standby.

Prerequisites Before You Start

Console or management access to the switch. If the management IP is unknown, you can reach the switch over the serial console or IPMI to read it.
The switch firmware at a version that supports the embedded SM features you want; the CLI below is standard MLNX-OS.
Host HCAs with up-to-date firmware and the opensm utilities installed for testing (ibstat, ibping, iblinkinfo from the infiniband-diags package).
Physical-layer sanity: cables seated correctly, correct cable type for the port (copper vs optical), and — if you use splitters — the breakout configuration matching the intended port split.

Step 1: Find the Switch IP Address

If you do not know the switch's management IP, perform an ARP scan of your management subnet to discover it (look for the Mellanox switch's MAC/vendor OUI in the results):

sudo arp-scan --localnet

If arp-scan is not installed, install it with apt install arp-scan on Debian/Ubuntu. The output lists each responding host with its MAC address and vendor; Mellanox interfaces typically show as Mellanox Technologies or a similar OUI. If the switch was never given a static address it may be using DHCP — check the DHCP leases on your management server, or read the address from the switch console.

Step 2: Enable the Subnet Manager via CLI

SSH into the switch (default username/password is admin / admin), then enable the configuration terminal and turn on the subnet manager:

$ ssh admin@10.1.10.136
switch-hostname > enable
switch-hostname # configure terminal
switch-hostname (config) # ib smnode my-sm enable

Walking through the sequence: enable moves you from user EXEC mode into privileged mode, configure terminal enters global configuration, and ib smnode my-sm enable switches on the embedded subnet manager instance named my-sm. The name is arbitrary but is what you will reference in later commands. Your subnet manager is now running. Use show ib sm to verify the SM state and check that port lights transition from amber to green as links activate.

A useful follow-up command shows the SM nodes defined on the switch and their state:

switch-hostname # show ib smnode my-sm
switch-hostname # show ib sm
switch-hostname # show ib smnode my-sm sm-info

Once the SM is enabled, the switch begins sweeping the fabric immediately. Ports that were stuck in the Initialize or Down state move through Arm to Active as the SM programs each connection.

Step 2 (Alternative): Enable via the Web Console

Navigate to the switch's IP over HTTPS (e.g. https://10.1.10.136).
Log in with username admin / password admin.
Open the IB SM Mgmt tab in the top navigation bar, then click Base SM.
Check the box next to the switch name and the SM Enabled checkbox.
Click Apply.

Confirm success by looking at the top-right of the screen: it should read Subnet Manager is running.

Verifying the Fabric

Once the subnet manager is active, links between the switch and connected HCAs (host channel adapters) should come up. Check link states on the switch with the show ib ports command and verify end-to-end connectivity between hosts with ibping or ibstatus on the hosts themselves. For deeper MLNX-OS configuration and troubleshooting, see the MLNX-OS user manual referenced in the original tutorial.

A practical verification sequence looks like this. On the switch:

switch-hostname # show ib ports
switch-hostname # show ib sm
switch-hostname # show ib smnode my-sm sm-info

On any compute node with an HCA:

# Per-port link state, width and speed
ibstat

# Dump the fabric as the SM sees it
iblinkinfo

# Ping another HCA's LID to prove end-to-end pathing
ibping -S -C mlx5_0            # server side
ibping -c 5 -L <remote-LID>    # client side

ibstat should report State: Active, Physical state: LinkUp, and the expected width (4x) and speed (EDR = 100 Gb/s per port for a 4x link). If ibstat shows the link up but iblinkinfo reports no LIDs, the SM is not sweeping that port — revisit Step 2.

Reading Port States and LEDs

InfiniBand ports move through a well-defined state machine, and the LED colors tell the same story from the physical side:

Down / amber. No logical link. Usually the SM is not running, the cable is faulty, or the far end is down.
Initialize (amber blinking). Physical link is present; the SM is configuring the port. A port sitting here forever means the SM is not reaching it.
Arm. The port is configured but awaiting the full path-table setup; a brief transient state normally.
Active (green). Fully configured, forwarding, with a valid LID. This is the state you want on every connected port.

Common Problems and Fixes

SM enabled but links stay amber. Check that the port is not administratively shut, that the cable and transceiver are seated, and that the HCAs are powered. Run show ib ports and look for ports in Down rather than Initialize.
SM does not survive a reboot. Configuration changes on MLNX-OS are persisted with configuration write; a setting applied but never saved is lost on restart. Enter config mode and write the running configuration.
Two SMs fighting. If an OpenSM instance on a host and the switch SM have the same priority, behavior can flap during failover. Give one of them a clearly higher or lower priority and document which is intended to be master.
Link comes up at reduced width or speed. A 4x link training at 1x or at a lower generation usually means a bad lane, an unsupported cable, or a breakout split that does not match the port configuration.
Traffic works but performance is poor. Verify MTU consistency across the fabric and confirm that the SM’s routing is using the paths you expect; in fat-tree fabrics a single misprogrammed switch can push everything down one uplink.

Useful MLNX-OS InfiniBand Commands

show ib sm                                  # subnet manager state
show ib smnode                              # configured SM nodes
show ib smnode my-sm sm-info                # details of one SM instance
show ib ports                               # port states, width and speed
show ib port <n>                            # one port in detail
show ib smport <n>                          # SM view of a port
show ib counters <n>                        # error and traffic counters
show interface ib <n>                       # link status from the interface side

Bookmark show ib counters: rising symbol-error or link-down counters on a specific port are the fastest signal that a cable or transceiver is degrading, long before users notice.