ONIE and Onyx (MLNX-OS) Install

原文：ONIE and Onyx (MLNX-OS) Install — theDXT (Daniel Keer)

Switches that support ONIE (Open Network Install Environment) are amazing switches because you can just change which NOS (Network Operating System) you are running relatively easily.

ONIE was created by Cumulus Networks in 2012. In 2020 Nvidia bought Cumulus just after purchasing Mellanox the year before.

I will detail step-by-step how to install ONIE and how to install the Onyx (MLNX-OS) NOS on the SN2410 switch. The process for other NOS and other switches should be similar.

What ONIE Is and Why It Matters

ONIE - the Open Network Install Environment - is a tiny Linux environment that ships preinstalled in the flash of a "bare metal" switch. It does exactly two jobs: it lets the operator install an arbitrary network operating system onto the switch without a vendor-specific recovery tool, and if that NOS is ever wiped or corrupted, it comes back and offers to install another one. Before ONIE, changing the OS on a switch meant vendor-locked images, licensing gymnastics and a recovery process that assumed you had the original vendor's support contract. After ONIE, the switch chassis is closer to a server with a UEFI firmware - the OS is a replaceable layer.

ONIE was created at Cumulus Networks in 2012 and contributed to the Open Compute Project; NVIDIA acquired Mellanox in 2019 and Cumulus Networks in 2020, which is why the same codebase now sits under an NVIDIA banner and why Mellanox/NVIDIA Spectrum switches (like the SN2410 used throughout this walkthrough) are among the best-supported ONIE platforms in existence. The same mechanism runs on Dell, Edgecore, Celestica and Quanta hardware, and it is what makes modern multi-vendor leaf-spine designs practical: identical switches can run Cumulus Linux, SONiC, Onyx, or a vendor NOS depending on what the fabric needs.

Functionally, ONIE boots into a menu with a handful of options. Install OS boots ONIE and waits for an installer to be fetched; ONIE: Embed ONIE refreshes the ONIE image in flash, which is what you do when the onboard ONIE is old or damaged; Rescue gives you a shell for manual recovery; Uninstall OS wipes the NOS and returns the switch to the discovery state. When ONIE has no NOS to boot and no local installer, it enters discovery mode: it brings up every interface, tries DHCP on each, and keeps polling for an installer advertised by the network (via DHCP options or a zeroconf-style lookup). That behaviour is convenient in a provisioning rack and deeply annoying on a bench, which is exactly why the very first thing this procedure does after selecting the install option is stop discovery.

The NOS installer itself is vendor-authored but ONIE-shaped: a single executable file that ONIE runs, which partitions the SSD, unpacks the filesystem and installs the bootloader. That is why the file extension varies - .bin for Onyx, .bin or a signed bundle for SONiC, an image file for Cumulus Linux - and why a wrong-platform installer will either refuse to run or produce a switch that boots to nothing. Always match the installer to the exact platform model and ASIC generation.

Prerequisites

• Compiled ONIE recovery image for your switch.

I need the one for Mellanox/Nvidia that file will have a name similar to this onie-recovery-x86_64-mlnx_x86-r0.iso

• The NOS install file.

I’ll be installing Onyx, the Onyx install file will have a name similar to this one X86_64-3.9.3202-installer.bin if you google around you should be able to find it.

• Console connection to the switch
• USB drive
• Network cable plugged into mgmt0 on a network with DHCP.
• BIOS password if one is applied. Here’s how to reset the BIOS password for Onyx (MLNX-OS) switches. If the SSD in the switch has nothing on it then you can get by without the BIOS password.

Installing ONIE

• Download the most recent version of Rufus.

• Write the ONIE recovery image to the USB drive.

The default settings should be fine. This is what I used.

Rufus Settings

If Rufus detects ISOHybrid then select Write in ISO Image mode.

Rufus ISOHybrid detected

• Plug the USB drive into the switch

• Connect to the console of the switch

The Console Settings are:
* Speed:115200
* Data bits:8
* Stop bits:1
* Parity:None
* Flow control:None

• As the switch boots press Ctrl+B to enter BIOS and enter the password admin or your custom BIOS password.

MLNX BIOS password

If there is nothing installed on SSD in the switch it will boot off USB even if you don’t have the BIOS password.

• Select the Save & Exit menuand select your USB drivefrom theBoot Override options. (don’t select UEFI as there have been known issues with some switches getting stuck in boot loops)

Selecting the non UEFI option to boot off the USB drive.

• Select ONIE: Embed ONIE

Selecting ONIE: Embed ONIE

• ONIE will begin installing

• Once the ONIE install is completed the switch will reboot.

Installing NOS

In my case I will be installing Onyx however these steps should be similar for other NOS.

• Select ONIE: Install OS

Selecting ONIE: Install OS

ONIE has a discovery mode that will keep looking for the install file, we need to tell it to stop doing that.

• Run the following command to stop the ONIE discovery onie-stop

Stopping ONIE discovery

Now we need to copy the NOS install file to the switch. ONIE support a few methods to do that such as HTTP, FTP, TFTP, and SCP.

I tried with TFTP but I found it to be very slow, I ended up using a quick HTTP server called HFS.

• Run the following company to copy the install file to/tmp/ wget http://IP_HTTP_Server/FOLDER/INSTALL_FILE.bin -P /tmp/

For me that command will look like this wget http://192.168.3.105/TFTP-Root/X86_64-3.9.3202-installer.bin -P /tmp/

Copying the Onyx install file to the switch

Now we need to install the NOS.

• Run the following command to install the NOS onie-nos-install /tmp/INSTALL_FILE.bin

For me that command will look like onie-nos-install /tmp/X86_64-3.9.3202-installer.bin

Installing Onyx on the switch

• Once the NOS install is completed the switch will reboot and load the NOS you just installed.

First Configuration After the NOS Boots

Once the switch reboots into Onyx, the console lands on a login prompt with the factory credentials admin / admin. Before the switch does anything useful you need a management address, a hostname and a saved configuration - and the ordering matters, because a switch that reboots without a saved config comes back to defaults. On Onyx the sequence looks like this:

# first login, then immediately change the password
switch login: admin
switch password: admin
switch > enable
switch # configure terminal
switch (config) # hostname leaf-01
switch (config) # interface mgmt0 ip address 192.0.2.20 /24
switch (config) # ip route vrf default 0.0.0.0 /0 192.0.2.1
switch (config) # exit
switch # write memory
switch # show version
switch # show interfaces mgmt0

The write memory at the end is not optional on any of these platforms, and it is the single most common reason a freshly installed switch "loses" its configuration after a power cycle. Confirm with show configuration and by comparing the running and saved state; on Onyx, configuration write in config mode is the equivalent verb if you prefer it.

ONIE Command Reference

ONIE's shell is busybox under the hood, and a small set of onie-* wrappers does the useful work. These are the ones worth knowing when something goes wrong at 2am:

Two habits keep installs fast. Bring the install file to the switch with wget over HTTP rather than TFTP - TFTP's 512-byte block acknowledgement turns a 400 MB image into an hour of waiting, while an HTTP server on the same bench moves it in under a minute. And once the installer is on the switch, run it from a booted ONIE session rather than relying on discovery, so you can see the partition and package messages on the console instead of guessing why a discovery install quietly stalled.

Choosing Where the Installer Comes From

The NOS image itself has to come from somewhere: for Onyx, the installer matching your switch model and the MLNX-OS release you intend to run. Downloads require a support entitlement in most cases, and mismatched builds are the usual cause of a switch that installs successfully and then refuses to come up. Keep the exact filenames in your build documentation - X86_64-<version>-installer.bin is Onyx's naming, and the platform prefix matters more than the version number when you are choosing what to download.

It is also worth keeping the ONIE recovery image for each platform in the same artifact store as the NOS images. Recovering a switch whose flash has been corrupted is a matter of writing the recovery image to a USB stick and repeating the ONIE install described above; without it, the same chassis becomes a support case. Version-control the SHA256 of both files alongside the filename so a later engineer can tell a good image from a truncated download.

Troubleshooting Common ONIE Problems

Boot loop instead of a menu: the classic cause is selecting the UEFI entry for the USB drive rather than the legacy one. Some switches get stuck cycling because the BIOS keeps re-selecting the wrong boot device; power off, re-enter the BIOS, and pick the non-UEFI USB entry. If the SSD has no NOS and no ONIE, the BIOS should fall through to USB, and this is the case where you can install without knowing the BIOS password.

Garbage on the console: the console speed is wrong. ONIE and the MLNX BIOS both speak 115200 8N1, but some terminal servers default to 9600. A screen full of noise and a screen with no output are the same symptom with the same fix.

Discovery finds nothing: expected on a lab bench with no provisioning server. Run onie-stop and install manually, or stand up a DHCP server that advertises the installer URL if you are provisioning a rack at once.

onyx-nos-install fails partway: almost always a truncated image - re-download and verify the checksum before blaming the switch. If the installer refuses to run entirely, check that the file is marked executable (chmod +x) and that you have enough free space on the SSD after a previous failed attempt; wiping the partitions with the ONIE uninstall option gives you a clean staging area.

Switch boots Onyx but cannot reach the network: the installer does not configure anything about your environment. Verify show interfaces mgmt0, check the default route, and remember that on a freshly imaged switch every data port is administratively enabled but unconfigured - nothing will forward until you build a configuration.

Provisioning a Rack: DHCP-Driven ONIE Discovery

Installing one switch by hand is fine; installing twenty is not. ONIE discovery exists for the rack-scale case: it brings up every interface, asks for DHCP, identifies itself with a vendor-class string derived from the platform (the onie-* value onie-sysinfo prints), and looks for an installer URL supplied by the provisioning server. To use it, do not run onie-stop. Instead stand up a DHCP scope that answers ONIE and points it at an HTTP server holding the exact installer for each model:

subnet 192.0.2.0 netmask 255.255.255.0 {
  range 192.0.2.100 192.0.2.199;
  option routers 192.0.2.1;
  # ONIE sends a vendor-class identifier beginning with "onie_"; match it
  class "onie" {
    match if (option vendor-class-identifier ~= "^onie_");
    option default-url = "http://192.0.2.10/onie-installer";
  }
}

The URL must serve a file ONIE can execute, and the platform-specific name matters: ONIE looks for a filename matching the platform and falls back to a generic onie-installer, so a rack containing two switch models needs two URLs (or a web server that serves the right file per user agent). This is the same mechanism behind zero-touch provisioning workflows that render a configuration onto a switch as it comes out of the box. If discovery runs and finds nothing, nothing is broken — ONIE is simply polling a network that has no answer for it, which is why on a bench the right move remains onie-stop followed by a manual onie-nos-install.

Updating ONIE Itself

The ONIE in flash can be older than the NOS you want to install, and that mismatch produces failures that look like a bad NOS image. Before installing a NOS on a switch whose ONIE has never been touched, check the version and update it when the release notes require a minimum:

ONIE:/ # onie-sysinfo
ONIE:/ # onie-self-update http://192.0.2.10/onie-updater-x86_64-mlnx_x86-r0

The self-update runs from a booted ONIE session, writes the new ONIE to flash and reboots into it. The USB route — selecting ONIE: Embed ONIE with a recovery image on a stick — is the fallback when the flash copy is damaged or the switch will not stay up long enough for a network update. Keep both the recovery ISO and the updater for every platform you own, because they are not interchangeable: the recovery image writes ONIE to flash from scratch, the updater replaces it in place, and the wrong file for your model fails in a way the console only explains if you read the first lines of output.

Switching NOS Later, and Rolling Back

The point of ONIE is that the NOS is replaceable, and doing it in place takes a menu selection and a stop command:

ONIE:/ # onie-nos-mode -r          # force the next boot back into ONIE recovery
ONIE:/ # onie-nos-mode -s          # force the next boot into NOS mode
ONIE:/ # onie-uninstaller          # remove the running NOS and return to discovery

In practice you switch NOS by rebooting into ONIE, stopping discovery, fetching the new installer over HTTP and running onie-nos-install exactly as in the first install. Two cautions. The new NOS will not reclaim the old one’s configuration, so export the running configuration first and treat the switch as new hardware. And when an installer expects a particular ONIE version or secure-boot state it stops early rather than writing a half-finished disk; read the first screen of output for the reason, update ONIE, and retry instead of repeating the install blindly.

Related Reading

For the original walkthrough of the same process on a related platform, see ONIE and ONYX (MLNX-OS) install guide. Once the switch is up, upgrading Onyx (MLNX-OS) covers the in-place upgrade path, and getting started with Mellanox switches takes the first configuration further. Backups belong in Git from day one - see backing up MLNX-OS configuration to a server - and the ONIE-based install route on other hardware is covered in our Dell OS10 ONIE vs in-place image install and SONiC quick start via ONIE guides.

That is all it takes to install ONIE and NOS onto a switch.

If you want to read more about ONIE you can do so on their website here https://opencomputeproject.github.io/onie/