SONiC Troubleshooting: Packet Drops and Link Issues

SONiC gives you Linux-grade tooling on open networking switches, but its counter model and CLI differ from traditional NOSes — which is exactly where troubleshooting goes wrong. This guide, based on the official SONiC community troubleshooting wiki, explains how to interpret interface counters, read transceiver DOM data, generate techsupport dumps and safely isolate a suspect device from the network. It also covers what each counter family actually means at the ASIC level, so that the next command you type is the right one.

Investigating Packet Drops with Interface Counters

The first command in almost every SONiC investigation is the port counter table:

admin@sonic:~$ show interfaces counters
      Iface            RX_OK      RX_RATE    RX_UTIL    RX_ERR    RX_DRP    RX_OVR
-----------  ---------------  -----------  ---------  --------  --------  --------
  Ethernet0  471,729,839,997  653.87 MB/s     12.77%         0    18,682         0

In this sample the port is carrying a healthy 12.77% of line rate with no error counters at all, yet roughly 18,700 frames have been discarded on ingress. That combination — clean physical layer, non-zero drops — narrows the search dramatically and points at the forwarding pipeline rather than the cable.

Learn the counter vocabulary:

• RX_ERR/TX_ERR: physical-layer (L2) drops — FCS errors, runt frames. Non-zero values usually mean a physical link problem.
• RX_DRP: ingress pipeline drops (L2/L3/ACL) plus insufficient ingress buffer.
• TX_DRP: egress buffer congestion drops, including WRED.
• RX_OVR/TX_OVR: oversized packet counts.

Because these come straight from the ASIC SDK through SAI, they describe what the silicon did, not what the Linux host observed. A port can show zero traffic in ip -s link and millions of frames in the SONiC table — the SONiC table is the one that matters.

Turning Counters into a Diagnosis

The value of the table is in the combinations, not the individual numbers. Four patterns cover most real incidents:

• Errors climbing, drops flat. The physical layer is degraded. Go to optics, connector cleanliness, cable type, speed negotiation and FEC mode. Nothing in the forwarding configuration will fix it.
• Drops climbing, errors flat. The pipeline is discarding frames that arrive intact. Check ACLs, ARP and MAC tables, VLAN membership, MTU and routing before touching hardware.
• TX_DRP with high TX_UTIL. Egress congestion. Read queue counters to find which class of service is losing frames and whether a burst rather than sustained load is responsible.
• RX_OVR or TX_OVR. A size problem — a jumbo frame path where one end accepts a larger MTU than the other.

Take a baseline sample, clear the counters deliberately, then sample again. The delta over a known interval is the only figure that describes present behaviour; a counter accumulated over months of uptime will send you chasing faults that were fixed long ago.

admin@sonic:~$ show interfaces counters -p 5
admin@sonic:~$ show interfaces counters errors
admin@sonic:~$ sudo sonic-clear counters

Checking Optical Signal Strength

admin@sonic:~$ show interfaces transceiver eeprom Ethernet12 --dom

Read the RX/TX power values from the DOM output. As a rule of thumb optical receive power should stay above -10 dBm. Note that AOC and DAC cables do not expose channel monitor values, so a missing reading there is normal, not a fault.

Compare lanes as well as absolute figures. On a four-lane 100G optic, readings of -5.7, -4.6, -5.0 and -12.5 dBm tell a story that no single number can: three lanes agree closely, one is more than 6 dB darker, and that lane is the one generating FCS errors. The usual causes are a partial fibre break, a dirty connector in a single position of the MPO, or a transmitter lane beginning to fail. DOM also reports TX power, bias current and module temperature, which lets you distinguish an ageing laser from an airflow problem.

admin@sonic:~$ show interfaces transceiver presence
admin@sonic:~$ show interfaces transceiver info Ethernet12
admin@sonic:~$ show interfaces transceiver error-status

If receive power is comfortably above threshold on both ends and error counters still climb, the problem is almost never the optic. It is usually FEC negotiation or a rate mismatch, and both are visible in show interfaces status as the negotiated speed and FEC mode. Four-lane 100G ports and their counterparts must agree on RS-FEC; a mismatch produces a link that comes up, passes some frames, and then accumulates errors indefinitely.

Link State, Flapping and FEC

admin@sonic:~$ show interfaces status
admin@sonic:~$ show interfaces link-flap

Link flap history separates a genuinely unstable physical link from one that was simply shut down. Frequent flaps with clean optics usually indicate power or environmental issues, an incompletely seated module, or a far-end port being bounced by automation. Where the platform supports it, the transceiver error-status output exposes per-lane loss-of-signal and loss-of-lock events that corroborate the flap counts.

Congestion and Queue-Level Drops

TX_DRP is not a single phenomenon. On modern merchant silicon, congestion drops may be counted per queue rather than per port, which is why a port can report drops while its queue table appears quiet — or the reverse:

admin@sonic:~$ show queue counters Ethernet0
admin@sonic:~$ show priority-group headroom Ethernet0
admin@sonic:~$ show buffer_pool watermark

A microburst is the classic case: one second of averaging shows a modest 12% utilisation while a millisecond burst overruns the buffer and discards frames belonging to a loss-sensitive class. Look for drops concentrated in a single priority queue. If the affected queue is designed to be lossless and PFC is configured, check headroom consumption — exhausted shared headroom turns a lossless queue into a lossy one, and the fix belongs in buffer configuration. PFC storms that spread the problem across the fabric are covered in our SONiC PFC watchdog article.

Ingress Drops Without Errors

When RX_DRP climbs while RX_ERR stays at zero, work through the pipeline objects:

admin@sonic:~$ show acl table
admin@sonic:~$ show acl rule
admin@sonic:~$ show arp
admin@sonic:~$ show mac
admin@sonic:~$ show ip route

An ACL applied in the wrong direction, an aged-out ARP entry, a VLAN mismatch between the physical port and the SVI, an unresolved next hop and a uRPF check are all capable of producing silent drops with immaculate optics. The pattern to remember is that these faults black-hole traffic without generating a single error counter, because nothing is physically wrong — the frame is simply being discarded by policy or by a missing forwarding entry.

Collecting a Tech-Support Dump

admin@sonic:~$ show techsupport

This is SONiC's equivalent of show tech on other NOSes: it packages counters, logs and system state into an archive you can hand to the community or your vendor. Generate it before touching anything else — especially before isolating the device. The archive is written to /var/dump/<HOSTNAME>_YYYYMMDD_HHMMSS.tar.gz and contains interface details, routing and BGP state, transceiver readings, syslog and the running configuration.

admin@sonic:~$ show techsupport --since "2 hours ago"
admin@sonic:~$ ls -lh /var/dump/
admin@sonic:~$ tar tzf /var/dump/sonic_20260914_101500.tar.gz | head -40

Timing matters more than completeness. A dump taken at the moment of failure preserves transient state — half-built BGP sessions, ASIC error registers, near-empty buffers — that has already vanished by the time someone opens a ticket. Copy the archive off the switch immediately afterwards; a switch that later reloads takes the evidence with it.

Checking Services, Logs and Reboot Cause

Not every outage is a network fault. SONiC runs each subsystem in its own container, and a dead or thrashing container looks exactly like a protocol failure from the outside:

admin@sonic:~$ show services status
admin@sonic:~$ docker ps
admin@sonic:~$ docker logs --tail 100 bgp
admin@sonic:~$ show logging
admin@sonic:~$ show reboot-cause

Reboot cause is the command people forget. A switch that restarted itself after an SDK crash or a watchdog timeout will look like a power incident until you check, and the incident report will be wrong for as long as that guess survives. Pair it with a scan of the syslog for the moments immediately before the event.

Isolating a Suspect Device

When a SONiC device looks like it is dropping traffic, shut down its BGP sessions first — neighbor by hostname, by IP, or all at once:

admin@sonic:~$ sudo config bgp shutdown neighbor SONIC02SPINE
admin@sonic:~$ sudo config bgp shutdown neighbor 192.168.1.124
admin@sonic:~$ sudo config bgp shutdown all

This withdraws the routes so the rest of the fabric reconverges around the suspect switch, while leaving the configuration database untouched. Because nothing is written to CONFIG_DB, returning the device to service is a single mirror command rather than a reload or a configuration restore:

admin@sonic:~$ sudo config bgp startup all
admin@sonic:~$ sudo config bgp startup neighbor SONIC02SPINE
admin@sonic:~$ sudo config interface shutdown Ethernet0

Confirm from a neighbour that the prefixes really were withdrawn before you continue diagnosing, otherwise you are troubleshooting a device that is still in the forwarding path. If you only need to drain one port, the interface-level shutdown is the narrower instrument.

A Short Checklist to Work Through

1. Record show interfaces counters before changing anything, then clear and re-sample to get a clean delta.
2. Separate errors from drops; each family has its own branch of the diagnosis.
3. For errors, read DOM per lane, then verify speed and FEC agreement on both ends.
4. For drops, inspect queue and priority-group counters, then ACLs, ARP/MAC and routing.
5. Capture show techsupport while the fault is live and copy it off the box.
6. Check service status, container logs and reboot cause for a non-network cause.
7. Isolate with config bgp shutdown, verify reconvergence, and keep diagnosing off-path.

Where to Go Next

For a broader SONiC workflow, see installing SONiC via ONIE and the Supermicro SONiC configuration guide, which covers port breakout and the initial configuration that most drop investigations eventually depend on. If you want the counter vocabulary explained alongside the rest of the diagnostic commands, our SONiC troubleshooting guide walks the full workflow, and the transceiver power thresholds reference explains how to read Rx and Tx dBm values correctly.

原文链接：https://github.com/sonic-net/SONiC/wiki/Troubleshooting-Guide