ONIE and ONYX (MLNX-OS) Install Guide for Mellanox Switches

Switches that support ONIE (Open Network Install Environment) are amazing switches because you can just change which NOS (Network Operating System) you are running relatively easily.

ONIE was created by Cumulus Networks in 2012. In 2020 Nvidia bought Cumulus just after purchasing Mellanox the year before.

I will detail step-by-step how to install ONIE and how to install the Onyx (MLNX-OS) NOS on the SN2410 switch. The process for other NOS and other switches should be similar.

This walkthrough is the complete flow, in the order you actually perform it: prepare the media, install ONIE, install the NOS with onie-nos-install, verify the result from the switch itself, and then run the first ONYX configuration. It also covers what to do when the install fails, because the failure modes here are almost always the same handful of things.

Prerequisites

• Compiled ONIE recovery image for your switch.

I need the one for Mellanox/Nvidia that file will have a name similar to this onie-recovery-x86_64-mlnx_x86-r0.iso

Take the newest build you can get rather than the newest file you happen to have on a USB stick: ONIE recovery images are built per platform family, and the x86_64-mlnx_x86-r0 string identifies both the CPU architecture and the platform. On Spectrum switches (SN2010, SN2100, SN2410, SN2700 and the rest of the SN2xxx/SN3xxx families) that same mlnx_x86-r0 recovery image is the correct one, which is why one USB stick covers an entire mixed leaf-spine cabinet. Download it from the ONIE project page or from NVIDIA's support site, and note that ONIE recovery images are published both as ISO (for USB install) and as a .bin (for in-band updates); the ISO is what you want here.

• The NOS install file.

I’ll be installing Onyx, the Onyx install file will have a name similar to this one X86_64-3.9.3202-installer.bin if you google around you should be able to find it.

The version in that filename — 3.9.3202 — matters more than usual, because everything after the first boot (licences, supported optics, feature set) is tied to it. Pick an ONYX release from the "recommended" or "long-term support" list for your platform, and check the release notes for the minimum ONIE version it expects. Installing a NOS that assumes a newer ONIE than the switch has is a classic source of "the installer starts and then stops".

• Console connection to the switch
• USB drive
• Network cable plugged into mgmt0 on a network with DHCP.
• BIOS password if one is applied. Here’s how to reset the BIOS password for Onyx (MLNX-OS) switches. If the SSD in the switch has nothing on it then you can get by without the BIOS password.

Three practical notes. The USB drive will be wiped completely, so use a spare (8 GB or larger). The console cable matters more than people expect — a cheap USB-serial adapter that cannot hold 115200 baud shows a boot menu full of garbage, and the fix is a different adapter, not a different switch. And DHCP on mgmt0 is convenience, not necessity: with no DHCP server you assign an address by hand inside ONIE, which is covered below.

What ONIE Actually Does at Boot

ONIE is a small Linux environment that lives in the switch's flash alongside the NOS. It is not the NOS and it is not a bootloader in the traditional sense — it is a recovery and installation platform that the bootloader starts when there is no valid NOS, or when you ask for it deliberately. Understanding its menu is the difference between a ten-minute install and an afternoon of guessing.

Note that there are two menus. The first is the BIOS/bootloader menu, where Ctrl+B lets you choose a boot device — that is where you point the switch at the USB stick. The second is ONIE's own menu, which appears once the USB recovery image has booted. Mixing them up is why people report that "ONIE's menu does not have the option I need".

Installing ONIE

• Download the most recent version of Rufus.

• Write the ONIE recovery image to the USB drive.

The default settings should be fine. This is what I used.

If Rufus detects ISOHybrid then select Write in ISO Image mode.

On Linux or macOS the same job is one dd command, and it is worth knowing because it also lets you script a batch of USB sticks:

# macOS: list disks, then write the ISO raw to the whole device
diskutil list
diskutil unmountDisk /dev/disk2
sudo dd if=onie-recovery-x86_64-mlnx_x86-r0.iso of=/dev/rdisk2 bs=4m && sync

# Linux
sudo dd if=onie-recovery-x86_64-mlnx_x86-r0.iso of=/dev/sdb bs=4M status=progress && sync

Write to the raw device (/dev/rdiskN on macOS, /dev/sdX on Linux), never to a partition. Double-check the device name with diskutil list or lsblk before pressing Enter — dd does exactly what you tell it.

• Plug the USB drive into the switch

• Connect to the console of the switch
  The Console Settings are:
  • Speed: 115200
  • Data bits: 8
  • Stop bits: 1
  • Parity: None
  • Flow control: None

• As the switch boots press Ctrl+B to enter BIOS and enter the password admin or your custom BIOS password.

If there is nothing installed on SSD in the switch it will boot off USB even if you don’t have the BIOS password.

• Select the Save & Exit menu and select your USB drive from the Boot Override options. (don’t select UEFI as there have been known issues with some switches getting stuck in boot loops)

The UEFI warning in that step is not superstition. Some Spectrum platforms have a known interaction between the UEFI boot path and the ONIE image that leaves the switch cycling through the boot menu instead of starting the installer. Booting the legacy entry costs nothing and avoids the loop entirely.

• Select ONIE: Embed ONIE

• ONIE will begin installing

• Once the ONIE install is completed the switch will reboot.

Embedding ONIE takes a few minutes and the console goes quiet for part of it — resist the urge to power-cycle. When the switch comes back it presents the ONIE menu instead of the BIOS menu, which is the sign that ONIE is now in flash. Leave the USB stick plugged in; the NOS installer is fetched over the network next.

Installing NOS

In my case I will be installing Onyx however these steps should be similar for other NOS.

• Select ONIE: Install OS

ONIE has a discovery mode that will keep looking for the install file, we need to tell it to stop doing that.

• Run the following command to stop the ONIE discovery onie-stop

That discovery mode is worth understanding before you fight it: in ONIE: Install OS the environment keeps looking for an installer advertised by DHCP, mDNS or a USB stick, and a stale installer URL left over from a previous project will be used if it finds one. onie-stop ends the search so the session stays in a shell and you stay in control of what gets installed.

Now we need to copy the NOS install file to the switch. ONIE support a few methods to do that such as HTTP, FTP, TFTP, and SCP.

I tried with TFTP but I found it to be very slow, I ended up using a quick HTTP server called HFS.

Before fetching anything, confirm the management interface is up and reachable:

# inside ONIE
ip addr show            # is eth0/mgmt0 up with a DHCP lease?
ip route show
ping -c 3 192.168.3.105 # can the switch reach the host serving the image?
ntpdate pool.ntp.org     # optional, but avoids TLS/date oddities with HTTPS sources

If DHCP gave the switch nothing useful, set the address by hand — ONIE's shell is a normal Linux prompt, and whatever address you configure here is temporary and disappears with the reboot:

ifconfig eth0 192.168.3.20 netmask 255.255.255.0 up
route add default gw 192.168.3.1

• Run the following company to copy the install file to /tmp/ wget http://IP_HTTP_Server/FOLDER/INSTALL_FILE.bin -P /tmp/

For me that command will look like this wget http://192.168.3.105/TFTP-Root/X86_64-3.9.3202-installer.bin -P /tmp/

ONYX installers are around 300–400 MB, so a 100 Mbit management link needs a couple of minutes at most while TFTP can take twenty. Check ls -lh /tmp/*.bin against the vendor-published size and checksum: a truncated download produces an installer that starts and then fails partway through, which is far harder to diagnose than a failed wget.

Now we need to install the NOS.

• Run the following command to install the NOS onie-nos-install /tmp/INSTALL_FILE.bin

For me that command will look like onie-nos-install /tmp/X86_64-3.9.3202-installer.bin

onie-nos-install does the real work: it validates the installer, unpacks the platform-specific image, writes it to the internal SSD partition set, then hands off to the NOS's own installation scripts. Expect a short validation phase, a longer write phase with progress output, and a reboot. Anything else — an immediate "not a valid image" style error, or a write that stalls at a fixed percentage — is covered in the troubleshooting table below.

• Once the NOS install is completed the switch will reboot and load the NOS you just installed.

That is all it takes to install ONIE and NOS onto a switch.

If you want to read more about ONIE you can do so on their website here https://opencomputeproject.github.io/onie/

Verifying the Install

The reboot into ONYX is the point where people stop checking. Ten minutes of verification now saves a return trip later, because an install that "came up" but is missing a licence or running the wrong image is a problem that will find you at the worst possible moment.

switch login: admin / admin          # first boot default credentials
switch > enable
switch # show version                # ONYX image and build
switch # show system                 # uptime, platform, serial
switch # show inventory              # chassis part number and serial number
switch # show licenses               # is the licence present and valid?
switch # show bootvar                # which image partition will boot next?
switch # show interfaces ethernet status | include down

What to confirm, and what it tells you:

• show version reports the build you installed. If it reports the old one, the switch booted from the other partition — check show bootvar.
• show licenses shows the feature set you paid for. Licences are per-serial, so a licence from another switch will not work.
• show inventory matches the switch you think you are configuring — essential when commissioning several chassis at once.
• The management address answers over SSH, and no port shows down unexpectedly after a fresh install.

ONYX First Configuration After the Install

A freshly installed switch has a bare configuration: a hostname of switch-1, DHCP on the management port, and default credentials. Since software 3.8.2000 the admin and monitor passwords must be typed manually — there is no automatic default — so the wizard is mandatory rather than optional.

The guided path is configuration jump-start, which asks the same questions as the first-boot wizard and can be re-run at any time:

switch > enable
switch # configure terminal
switch (config) # configuration jump-start

# the wizard prompts, in order:
#   Hostname [switch-1]
#   DHCP on mgmt0 [yes]        - answer no to enter a static address
#   Enable IPv6 [yes]
#   Update time / NTP
#   Enable password hardening [yes]
#   Admin password             - must be typed
#   Monitor password           - must be typed

If you prefer to configure it explicitly — which is what you want when scripting or when following a build document — the equivalent commands are:

switch (config) # hostname core-sw-01
switch (config) # interface mgmt0 ip address 10.10.10.20 /24
switch (config) # ip route default gateway 10.10.10.1
switch (config) # ip name-server 10.10.10.53
switch (config) # no dhcp relay           # only if you run relays on this box
switch (config) # license install <license-key>
switch (config) # configuration write     # save, or lose everything at the next boot

Two ONYX behaviours catch people out here. configuration write is the equivalent of Cisco's write memory: without it a reboot reverts to the previous state, and it must be re-run after installing a licence or the licence disappears at the next power cycle. And the CLI is strictly tiered — standard, enable, config — so you cannot go from a read-only prompt straight into configuration mode.

Once the management plane is up, the useful next steps are the ones that make the switch maintainable: set the time source, configure logging to a central server, add an SNMP user, and save again. The full command walkthrough for those — time source, syslog target, SNMP user, and the port and VLAN configuration that follows — is listed in the related reading at the end of this article.

Troubleshooting the ONIE and ONYX Install

Recovery: Getting Back to a Known State

ONIE lives separately from the NOS, so recovery is a repeat of the install rather than a rebuild:

• Broken NOS configuration, working ONIE: boot into ONIE: Rescue and use onie-uninstall-style removal options, or simply reinstall over the top with onie-nos-install. ONYX's own factory reset and configuration management commands are covered in the MLNX-OS CLI troubleshooting article linked at the end of this page.
• Wiped or corrupted ONIE: the switch will refuse to boot anything useful. Put the ONIE recovery ISO back on a USB stick, boot it, and re-run ONIE: Embed ONIE from the start of this guide.
• Switch returned from a lab or another owner: ONIE: Uninstall OS clears the NOS and residual configuration; do not rely on a NOS-level factory reset alone, because log files and licence state can survive it.
• Different NOS on the same hardware: the same procedure that installs ONYX installs SONiC, Cumulus Linux or another compliant NOS — the differences are in the NOS's own first-boot workflow, not in ONIE.

Related reading

• Mellanox Switch Getting Started: First-Time Setup
• Mellanox Switch CLI: Getting Started with MLNX-OS
• MLNX-OS CLI Troubleshooting: Filters and Logs
• Dell OS10 Firmware Upgrade: ONIE vs In-Place Install