MLNX-OS CLI Troubleshooting: Filters, watch and Log Commands

NVIDIA (Mellanox) MLNX-OS switches come with an industry-style CLI that hides some of its most useful troubleshooting power in the pipeline operators: filtering, live monitoring and structured output. Once you learn that every show command can be filtered with include/exclude, watched at a refresh interval, or printed as JSON, log analysis and fault hunting on Spectrum and InfiniBand switches become dramatically faster. This guide covers the CLI modes, the show-command pipeline tricks and the logging commands that matter during troubleshooting.

CLI Modes

MLNX-OS has three main privilege levels. Commands are lowercase and case-sensitive:

• Standard mode: the initial shell; read-only queries of a restricted state set.
• Enable mode: enable - full state visibility plus actions such as reboot, but no configuration changes.
• Config mode: configure terminal - full configuration access, allowed for admin-role accounts. Exit back with exit or no configure.

You cannot jump directly from Standard to Config mode; you must pass through Enable.

switch > enable
switch # configure terminal
switch (config) #

The prompt tells you where you are, and that matters more on MLNX-OS than on most platforms, because the same keyword changes meaning with the context. show works in every mode, but configuration commands such as logging or interface exist only inside config mode, and the read-only commands available in Standard mode are a deliberately reduced set used for first-line triage by operators who should not be able to change anything. When a command appears not to exist, the first thing to check is the prompt - a missing command is nearly always the wrong mode rather than a missing feature.

Two habits save keystrokes throughout. Tab completion and the question mark work everywhere, and on a platform where some commands are long and hyphenated, they are the reliable way to discover the exact spelling. And because commands are case-sensitive, show version is not the same token as show Version; copying from a document that capitalised something is a surprisingly common cause of "command not found".

Filtering Show Output with include and exclude

Every show command can be piped to a filter, which saves you from scrolling through hundreds of lines during an incident:

switch (config) # show module | include "SW1|OK"
switch (config) # show interfaces | exclude "0"
switch (config) # show version | include version ignore-case
switch (config) # show asic-version | include SIB2

Filters support extended regular expressions, the ignore-case option, and the next/prev parameters to print surrounding lines. Multiple filters can be chained with additional pipe characters.

switch (config) # show interfaces ethernet 1/1 | include "link|speed|MTU"
switch (config) # show logging | include "log file|syslog|level"
switch (config) # show interfaces | include "1/17" next 3
switch (config) # show interfaces counters | include "1/17|1/18" ignore-case

The next and prev operators are the ones people rarely know about and use constantly once they do. A filter that matches only the interesting line strips the context that explains it - the interface name above, the counters below. Asking for a few lines after each match keeps a compact output while preserving the structure. Chain a second filter onto the result when the match is still too broad, and build the pattern with alternation rather than running the same command three times.

Quoting is not optional. Patterns are handed to the regular-expression engine, so a bare pipe or space will be parsed as syntax rather than as part of the pattern; wrap the whole expression in double quotes. That single rule removes most failed filter attempts.

Live Monitoring with watch

The watch operator re-runs a show command at a fixed interval (2 seconds by default) and updates the screen, which is perfect for watching counters while you reproduce a fault:

switch (config) # show interfaces counters | watch interval 5
switch (config) # show power | watch diff interval 1

The diff keyword highlights what changed between two consecutive iterations. Exit watch mode with Ctrl+C. watch cannot be combined with json-print.

Watch turns a static snapshot into a measurement. A counter that reads 4,000 discarded frames tells you very little; the same counter watched while you unplug and re-seat a cable tells you exactly when the problem started and how fast it grows. Two practical notes: keep the interval long enough that the output is readable (one second is usually too fast on a busy switch, five is comfortable), and pick a filter that keeps the screen short, because watch refreshes the whole view and a hundred-line output will scroll past the one line you care about.

Structured Output with json-print

For automation and precise parsing, pipe any show command to json-print to get the output as JSON instead of a formatted table:

switch (config) # show system profile
Profile: eth-single-switch
switch (config) # show system profile | json-print
{ "Profile": "eth-single-switch" }

switch (config) # show interfaces ethernet 1/1 counters | json-print
switch (config) # show inventory | json-print
switch (config) # show module | json-print

The value is not for humans - it is for scripts. Formatted CLI output changes with firmware versions and shifts columns with the length of values; JSON keys stay stable. When you automate an inventory collection or a health check that feeds a monitoring system, | json-print removes the brittle text parsing that otherwise breaks on every upgrade. Remember the one restriction: json-print cannot be combined with watch, so a continuous monitor and a machine-readable dump are separate commands.

Logging Commands for Troubleshooting

Event and log inspection is the core of MLNX-OS troubleshooting. The key commands:

show logging

Displays the logging configuration: local and remote log levels, number of archived log files (default 10), rotation size threshold, and configured remote syslog servers.

show log

Shows the log file itself with paging; supports continuous (tail -f style), file selection and regex matching:

switch # show log
switch # show log files 3
switch # show log matching "link down"
switch # show log continuous not matching "heartbeat"

The log-file commands are where a slow-burning fault gets found. show log files lists the rotated copies, so you can walk backwards through them when the incident began before the current file. show log matching takes a regular expression, and the continuous modifier follows the file live. Combining the two - following the log continuously while excluding a noisy recurring message - is the fastest way to watch a flapping link without drowning in unrelated events:

switch # show log files
switch # show log continuous matching "1/17"
switch # show log continuous not matching "heartbeat"

Remote syslog is configured from config mode, and log files can be rotated or deleted manually:

switch (config) # logging 10.10.10.5 protocol tcp
switch (config) # logging files delete current
switch (config) # show logging events interfaces source-counters

Event source-counters output gives per-interface Rx/Tx discard, FCS error, symbol error and storm-control counters - a fast first stop when a port looks unhealthy.

Send logs to a remote collector before you need them, not during the incident. A local log on a switch that reboots with the fault is a local log you no longer have, and the rotation threshold can quietly discard the first minutes of an event if the device is otherwise busy. Adding a syslog server to every switch is a small configuration change with an outsized payoff the first time a switch reloads unexpectedly.

A Worked Troubleshooting Sequence

The pieces above combine into a repeatable method for the classic report - "the link to server X keeps dropping":

1. Get to Enable mode and confirm the platform state: show version, show module, show inventory.
2. Narrow to the port with a filter rather than reading the whole interface table: show interfaces | include "1/17" next 3.
3. Look at the deeper counters for that port: show interfaces ethernet 1/17 counters, then show logging events interfaces source-counters for discards and error classes.
4. Reproduce the fault while watching live: show interfaces counters | watch interval 5. Note which counter moves first.
5. Correlate with the log: show log continuous matching "1/17" in a second session, or show log files to page back if the event has already passed.
6. Only after the counter and the log agree on a cause, change something. Changing a port before you have both is guesswork with a maintenance window attached.

Steps 4 and 5 are the ones that make MLNX-OS pleasant to debug: the switch can show you the fault developing in real time and the log corroborating it, in the same terminal session, without a packet capture.

Log Settings to Fix Before the Incident

Troubleshooting speed is mostly decided before the fault happens. Three settings on MLNX-OS are worth standardising across a fleet:

switch (config) # logging 10.10.10.5 protocol tcp
switch (config) # logging 10.10.10.6 protocol udp
switch (config) # logging files rotation criteria size 20
switch (config) # logging files rotation max-num 20

The first two send events to a collector over TCP, which is reliable and survives a busy network, and UDP as a fallback - a combination that costs nothing and has saved many investigations. The rotation settings decide how much local history exists when the collector is unreachable; raising the maximum number of archives from the default 10 to 20 doubles the window you can search with show log files and show log files 3. Whatever you choose, know the number: the first question in a post-incident review is usually "how far back does the log go", and guessing wrong means the evidence has already been rotated away.

Keep the two log views distinct in your head. show logging reports the configuration - levels, rotation, destinations, and the count of available event types. show log and show log files print the records. When an operator says "logging is not working", the answer is almost always in the first command: either the level excludes the message, the file is full and rotating, or no remote server was ever configured.

Common Pitfalls That Waste Time

• Wrong mode. A configuration command issued from Enable mode fails with a syntax error rather than a permission error. Check the prompt before rewriting the command.
• Unquoted filter patterns. Regular expressions must be wrapped in double quotes; a bare pipe or space is parsed as CLI syntax.
• Case sensitivity. Command keywords are lowercase. Pasted documentation with capitalisation is a frequent source of "unknown command".
• watch plus json-print. The two cannot be combined. Run the continuous view for humans and the JSON dump for scripts, separately.
• Reading counters without a baseline. Absolute counter values mean little; compare before and after, or watch them live, before concluding a port is faulty.
• Filtering away the context. A single matched line without the interface name above it or the counters below it often looks like an error when it is not. Use next/prev.

Useful Verification Commands

show version
show inventory
show module
show system profile
show interfaces counters
show logging events
show logging
show log files

Related reading: MLNX-OS CLI filters, watch and link diagnostics, our MLNX-OS CLI modes and commands guide and Mellanox MLNX-OS VLAN and IP routing configuration.

Original article: NVIDIA MLNX-OS User Manual: Command Line Interface