SONiC PFC Watchdog: Detecting and Taming PFC Storms

Lossless Ethernet depends on PFC pause frames, and pause frames are exactly the mechanism that can freeze a whole fabric when one misbehaving device keeps its priority queue paused. The SONiC PFC watchdog exists for that case: detect the storm, mitigate it, then restore PFC when the buffer calms down. This article explains the three functional blocks and the CLI you need to run it in production.

Why PFC Storms Matter

PFC back-pressure is supposed to be local. When a downstream device keeps sending pause frames for a priority group that never drains, the pause propagates upstream port by port until the entire lossless path stops forwarding - including the storage traffic you built the lossless fabric for. This is the failure mode the watchdog is designed to interrupt.

The mechanism is worth spelling out, because the watchdog only makes sense once you see why it is safe. A PFC pause frame tells the link partner "stop transmitting priority 3 for a while". Normally that pause is short: the local queue drains, the buffer frees up, and a resume or a timeout ends the pause. A storm happens when the pause never lifts - either because the receiver's queue is genuinely stuck behind a blocked egress port, or because the receiver itself is misbehaving and holding the pause indefinitely. Because pause frames are honoured, the sender stops, its upstream neighbour fills its own buffer for that priority, that neighbour sends its own pause frames, and the back-pressure ripples outward. Within milliseconds a single stuck queue can stop a whole spine tier. TCP does not help, because no packet is being dropped for TCP to react to, and RoCEv2's congestion control never sees the congestion because nothing is being marked.

Detection, Mitigation, Restoration

• Detection - the watchdog monitors each lossless queue. If a queue stays in the paused state for longer than T0 (a port-level parameter, typically in the hundreds of milliseconds), a storm is declared. Detection is per port and disabled by default.
• Mitigation - two choices per queue. drop discards packets already in the queue, drops subsequent packets bound for it and stops generating pause frames for that priority group - the local port is sacrificed to save the fabric. forward ignores incoming pause frames and keeps forwarding. Drop is the default and the safer default for storage fabrics.
• Restoration - the watchdog counts PFC frames and, if none arrive for T1, re-enables PFC on the queue and returns reserved buffer to its original value. T1 is also port-level and on the order of hundreds of milliseconds.

Two details in that list are easy to miss and both matter. First, detection does not require the queue to be full - as soon as the duration a queue has spent in the paused state exceeds T0, the storm is declared, even if the queue is empty. A paused-but-empty queue is the earliest possible signal and the watchdog takes it. Second, the watchdog only ever operates on lossless queues: PFC has to be configured on the device and the queue has to be mapped to a priority group, otherwise there is nothing for the feature to watch.

The Three Timers, Precisely

The configuration knobs are few, which is precisely why choosing them badly is common. There are three numbers in play:

• T0, the detection time (--detection-time, stored as detection_time in the Config DB, in milliseconds). How long a queue must stay paused before a storm is declared. Too low and legitimate flow control gets treated as a fault; too high and the pause has already propagated across the fabric before anything happens.
• T1, the restoration time (--restoration-time, stored as restoration_time). How long the queue must go without receiving any pause frame before PFC is re-enabled and buffering is restored. Too low and the port flaps between drop and normal several times a second; too high and a recovered link stays degraded long after the event.
• The counter polling interval (the PFC_WD flex-counter group's POLL_INTERVAL). The watchdog reads PFC and queue counters from the ASIC on a timer; the polling interval is the resolution of everything above. A detection time of 200 ms cannot be enforced accurately if counters are polled every second.

A practical starting point for a RoCEv2 storage fabric is T0 in the 200-400 ms range with T1 equal to or slightly greater than T0, then tuning against the counters described below. Anything below 100 ms will start reacting to legitimate, brief flow-control pauses during a storage rebuild or a microburst.

Configuration and Verification

pfcwd show config
pfcwd show stats
pfcwd start --action drop ports Ethernet116 detection-time 300 --restoration-time 300
pfcwd stop Ethernet116

Tune T0 above the worst legitimate transient (microbursts during a storage rebuild) and below the point where the fabric stops being useful. Values in the 200-400 ms range are common; anything below 100 ms starts dropping legitimate flow-control pauses.

Newer sonic-utilities releases moved the same feature under the config namespace and added two conveniences worth knowing about:

config pfcwd start_default
config pfcwd big_red_switch enable

config pfcwd start_default brings the watchdog up with platform default settings on every eligible queue instead of enumerating ports by hand - useful on a fabric where you want protection everywhere and will tune later. Note that it does not exist in SONiC 202311 and earlier. config pfcwd big_red_switch enable is the emergency lever: it activates the watchdog on all ports and queues where PFC is configured, without waiting for a per-queue storm to be detected first. Reach for it during an active incident, not as a steady-state design.

Verify what you configured before you trust it. pfcwd show config prints the per-port T0, T1 and action that the switch actually accepted - which is the only authoritative answer, because a platform that does not support the feature in hardware will silently ignore a subset of what the CLI allowed you to type.

The Option Everyone Forgets: Config DB

Like everything else in SONiC, the watchdog configuration lives in the Config DB and can be written there directly, which is how you deploy it fleet-wide from automation:

PFC_WD_TABLE:Ethernet116
  detection_time  = 300      ; milliseconds, 1-3 digits
  restoration_time = 300     ; milliseconds, 1-3 digits
  action          = "drop"   ; or "forward"

FLEX_COUNTER_GROUP_TABLE:PFC_WD
  QUEUE_PLUGIN_LIST   = <lua plugins>
  POLL_INTERVAL       = 1000
  FLEX_COUNTER_STATUS = enable

Two consequences follow. First, if FLEX_COUNTER_STATUS is not enable, the counters the watchdog needs are not being collected and the feature will not behave - so a template that sets the action but not the counter group is a very common silent failure. Second, because the state is in the database, config save and config reload semantics apply: an unsaved pfcwd start disappears after a reboot, and a saved configuration can come back with a stale T0 you had already tuned away from. The config_db.json save and reload workflow is the right reference here.

Reading the Evidence

show queue counters Ethernet116
show pfc counters
show interfaces counters errors Ethernet116
show buffer_pool watermark

A stormed queue shows its pause counters climbing while the output queue stays full. Pair the queue view with the interface counters from the SONiC packet drop troubleshooting guide: TX_DRP rising points at egress congestion (the watchdog's territory), while RX_ERR points at the physical layer instead.

The watchdog keeps its own accounting, and these are the counters that tell you whether it is doing its job or just hiding a bigger problem:

• Queue deadlock counter - how many times this queue entered the stormed state. A number that only ever goes up, on one port, is a device that should be investigated or replaced.
• Queue restore counter - how many times PFC was re-enabled after T1 elapsed. Deadlock and restore counts should track each other; a large gap means queues are being declared stormed and never recovering cleanly.
• Packets dropped due to deadlock (Tx and Rx) - the cost of the mitigation. These are the storage writes you lost. Trending this number is the honest way to argue for a real QoS fix instead of more watchdog tuning.
• Packets transmitted during deadlock - the equivalent figure for forward action ports, and the reason forward is only appropriate where the traffic is not loss-sensitive.

The watchdog also logs. A Notice-level message is emitted when a storm is detected and another when the queue is restored, and the restoration message reports the dropped or forwarded packet counts for both the output queue and the ingress priority group. Those two messages, in order, are the cleanest way to reconstruct an incident timeline from syslog alone.

Drop or Forward? Choosing Per Port

The action is per queue, which means you can be surgical instead of picking one behaviour for the whole switch. The trade-off is simple and worth stating plainly:

• drop sacrifices the traffic on that queue to protect the fabric. Buffered packets are discarded, subsequent packets bound for the queue are discarded, and incoming packets for the corresponding ingress priority group - pause frames included - are dropped so the switch stops generating pauses of its own. The blast radius is one port's lossless queue; the rest of the fabric keeps working.
• forward stops honouring incoming pause frames and keeps transmitting everything, including what was already buffered. Nothing is dropped because of the watchdog, but the port is no longer lossless: a downstream device that was relying on flow control now sees overruns, and RoCEv2 will retransmit.

On a storage-facing lossless port, drop is almost always right: it degrades one flow instead of the fabric. forward makes sense where the port faces a device you trust more than the fabric, or where the lossless guarantee is nominal and you would rather overrun a congested egress than lose packets inside the switch.

Design Notes for Lossless Fabrics

The watchdog is a safety net, not a QoS design. You still need to map DSCP and priority to the right lossless queues, size the shared buffer, and configure ECN so that RoCEv2 senders back off before pause frames are needed at all - the mechanism and thresholds are explained in RoCEv2 lossless Ethernet with PFC and ECN. On the CLI side, the broader command set is collected in the SONiC CLI cheat sheet, and PFC watchdog defaults differ per platform, so validate with show platforms and your vendor's SONiC release notes before rolling the same settings fleet-wide.

A few design rules separate fabrics that survive a PFC storm from fabrics that do not:

• Do not enable the watchdog on every port blindly. Enable it on lossless ports, with a documented T0 that reflects the traffic pattern on that port. A single fleet-wide profile guarantees that some port somewhere is either unprotected or flapping.
• Tune T0 above your worst legitimate pause. Measure it: run a storage rebuild or a backup window with the watchdog disabled, watch how long the longest legitimate pause lasts, and set T0 above that with margin.
• Keep T1 short enough to recover, long enough to stop flapping. If the logs show repeated detect-restore pairs on the same queue, T1 is too low or T0 is too low - not the traffic.
• Treat a storm as a symptom, not an event. The watchdog interrupting a storm is a successful mitigation of a failure elsewhere: a NIC holding a pause too long, a mis-sized buffer, an oversubscribed egress port, or ECN thresholds set so high that pause frames are the first line of defence rather than the last.
• Do not run the watchdog as a substitute for DCQCN. PFC storms are what ECN and DCQCN are designed to prevent; the watchdog only cleans up when they fail. For the end-to-end tuning picture, see RoCEv2 lossless tuning with PFC, ECN and DCQCN.

What the Watchdog Does Not Do

Being clear about the boundaries prevents the two most common operational mistakes:

• It does not fix the cause. It converts a fabric-wide freeze into a single-port degradation. The storm will recur until the device or the buffer sizing that produced it is dealt with.
• It does not drop only the offending flow. The action is per lossless queue, not per flow or per priority. Every packet of that class on that port is affected, including well-behaved ones.
• It does not guarantee losslessness. Both actions break the lossless contract for some traffic; they differ only in where the loss lands.
• It is not a monitoring tool. On its own, the watchdog will not tell you a queue is trending badly. Alert on the deadlock counter grouped by port, and on any increase in the detect-restore rate.

Verification Checklist for a Change Window

1. Confirm PFC is configured and the queues you intend to watch are mapped to a lossless priority group.
2. Enable or confirm the PFC_WD flex-counter group; without it the counters the feature depends on are not collected.
3. Apply T0, T1 and the action per port, then read back with pfcwd show config - the switch's answer is the only one that counts.
4. Confirm the counters increment by generating a known-good PFC pause (a short, legitimate one) and checking that no storm is declared. If a legitimate pause trips the watchdog, T0 is too low.
5. Save the configuration and confirm it survives config save, so a reboot does not silently revert your tuning.
6. Baseline the deadlock and restore counters per port before the window closes, so the next incident has something to compare against.

Frequently Asked Questions

Is the PFC watchdog enabled by default in SONiC? No. Detection is disabled by default and must be enabled per port; the default action once you enable it is drop.

Why does the watchdog not fire on a congested port? Congestion alone is not a storm. The watchdog watches for a queue that stays paused past T0. If PFC is not configured on that port, or the queue is not lossless, the feature has nothing to observe.

Can I apply different settings to different ports? Yes - T0, T1 and the action are all per port, and pfcwd start accepts a port list so you can group ports that share a profile.

Does a storm always mean the watchdog will fire? Only if the storm lasts longer than T0. A burst of pause frames that clears inside T0 is exactly what PFC is supposed to do and should not trigger mitigation.

原文链接：https://github.com/sonic-net/SONiC/wiki/PFC-Watchdog