Junos ISIS Flap Case Study: Hello Padding and Fragmentation

This is a rare behind-the-scenes story from a service provider network: intermittent ISIS adjacency flaps on Juniper MX routers that resisted every obvious explanation — no CoPP drops, stable OSPF neighbors, healthy optics — until packet captures revealed the real culprit was hello padding and a 26-year-old MTU limitation inside Junos itself. The investigation ultimately led Juniper to introduce two brand-new CLI commands in Junos 22.4R3-S6. This article retraces the troubleshooting path so you can recognize the same signature in your own network, and it explains why a multi-vendor hello-padding mismatch can silently starve a healthy-looking routing protocol.

The Symptom: Simultaneous ISIS Adjacencies Down

Two Cisco NCS540 devices running SR-MPLS formed ISIS adjacencies with three Juniper MX routers (acting as BNGs) through a pair of QFX switches. Occasionally, one MX would lose all four ISIS adjacencies at once — a critical event, because the MX terminates PPPoE sessions over EVPN-MPLS VPWS, and a simultaneous flap disconnects all subscribers. The log signature looked like:

RPD_ISIS_ADJDOWN: IS-IS lost L2 adjacency to ncs540-2-cust on ae31.723, reason: Aged out
RPD_ISIS_ADJUP: IS-IS new L2 adjacency to ncs540-2-cust on ae31.723

The phrase reason: Aged out is the important detail. An adjacency does not age out because an interface bounced; it ages out because the hold timer expired without the required number of hello packets arriving. In other words, the link was up and forwarding the whole time — the protocol simply stopped hearing from its neighbour for roughly 30 seconds. That immediately reframes the problem from "a link is flapping" to "hello packets are being lost somewhere between the wire and the routing process."

Why Losing Four Adjacencies at Once Is the Real Clue

A physical-layer or optics fault is normally local: one member link degrades, one adjacency drops. Here, all four adjacencies on different logical interfaces went down in the same second. Four independent links do not fail simultaneously for physical reasons. That pattern points to a shared element — either a common transit device, or the router's own control plane. Because the same behaviour was seen on the QFX pair as well as directly attached links, the common element had to be inside the MX itself.

From a design standpoint, a BNG that terminates PPPoE over an EVPN-MPLS VPWS is unusually sensitive to control-plane interruption. Every lost adjacency triggers a routing recalculation that, in the worst case, tears down subscriber sessions, re-establishes them, and pushes a fresh storm of PADI/PADO authentication traffic back onto the control plane. This feedback loop — loss of hellos causing session churn, which causes more control-plane load, which causes more lost hellos — is what turned an occasional annoyance into a service-affecting outage. If you operate access networks, see our guide to PPPoE session establishment issues for the subscriber-side view of the same system.

Eliminating the Obvious

JTAC first suspected Control Plane Policing (ddos-protection), but there were no DDoS events and no ISIS/BFD drops. Packet captures on the QFX showed it received all ISIS packets, yet the MX control plane missed some — packets were being dropped between the Packet Forwarding Engine (PFE) and the Routing Engine (RE). Comparing show ddos-protection protocols isis statistics (PFE counters) with show isis statistics (RE counters) confirmed the gap:

show ddos-protection protocols isis statistics
show ddos-protection statistics
show isis statistics
show isis adjacency extensive
show log messages | match ISIS

The methodology is what matters here: pick one counter that is incremented where packets arrive (the PFE) and one that is incremented where packets are consumed (the RE). If the first is larger than the second, you have proven an internal drop, and you can stop blaming the optics, the fibre, and the peer device.

Root Cause: Hello Padding + Internal MTU 1500

Cisco enables ISIS hello padding by default, producing jumbo hello packets, while Juniper sends smaller packets. The PFE-to-RE link uses Juniper's proprietary TTP protocol (IP protocol 84) with a 26-byte header — and its MTU has been hardcoded to 1500 since the M40 in 1998. Anything larger gets fragmented, and under bursty BNG control-plane load, fragments timed out in the kernel reassembly queue, so three consecutive ISIS hellos were lost and adjacencies aged out. Setting Cisco's hello padding to hello-padding sometimes on one VLAN immediately stopped the flapping on that VLAN.

Hello padding exists for a legitimate reason: RFC 3719 recommends padding IIH packets to the link MTU so that an adjacency is never formed over a path that can pass small control packets but cannot pass full-size data packets. Padding protects against MTU asymmetry. The problem is that it also makes hellos large enough to trip a completely unrelated internal constraint on the neighbouring vendor's box.

Why Fragmentation, Not the MTU Itself, Broke the Hellos

It is tempting to think the fix is "raise the internal MTU," but the 1500-byte internal MTU had worked for decades because fragmentation normally repairs it transparently. What changed was load. Under normal conditions the ~1520-byte hello is fragmented into two pieces, both of which are reassembled long before the hold timer expires. Under sustained BNG control-plane load, the reassembly queue backs up, fragments are dropped when the queue is full, and the kernel's 60-second reassembly timeout keeps stale fragment entries alive longer than the ISIS hold time. The result is a protocol that loses consecutive hellos while every counter that operators normally check stays clean.

Capturing the Evidence on the Wire

Before JTAC narrowed the problem, the team did the one thing that settled the argument quickly: a span capture on the link between the NCS540 and the QFX, filtered on the ISIS protocol so the capture stayed small enough to keep running for hours. Two properties of a padded hello jump out immediately in such a capture. The first is length: instead of the roughly 1,500-byte maximum a normal ethernet frame allows, the padded hello arrives at the jumbo size the NCS540 is happy to emit because its own fabric MTU supports it. The second is fragmentation: on the receiving side the packet has been split into multiple IP fragments with sequential IDs, and the trace shows entire fragment sets disappearing rather than a steady trickle of single-fragment losses — the signature of a reassembly queue emptying under timeout rather than a congested link dropping packets at the tail.

Capturing simultaneously on both sides is the discipline that pays off. If the sender shows the padded hello leaving and the receiver shows only fragments arriving, you have localised the loss to the receiver's internal path without touching a single peer configuration. That is the exact point at which this diagnosis turned from "somebody's link is bad" into "our own PFE-to-RE path is dropping our own neighbour's hellos."

Alternatives Considered and Rejected

Three other fixes were on the table, and understanding why each was rejected explains why the reassembly-timeout change is the right one. Raising the internal TTP interface MTU would have fixed the symptom but touched a code path that had been stable since 1998 and would have required regression testing across every platform that shares it. Disabling hello padding on the Cisco side fixes only the cases where the operator controls both ends, and service providers rarely do — the customer edge is someone else's equipment. Raising the ISIS hold timer would have masked the problem by making the adjacency more tolerant of lost hellos, at the cost of slower failure detection when a link really does go down; in an access network where reconvergence speed is the whole point, that trade is a loss. Lowering the reassembly timeout by contrast changes nothing about the protocol, the marking, or the convergence behaviour — it simply stops a buffer that is no longer useful from occupying space the next hello needs.

The Fix: New Junos Commands in 22.4R3-S6

Rather than change the internal interface MTU (a 20-year-old code path), Juniper chose to lower the kernel fragment reassembly timeout from 60 to 10 seconds so the queue flushes faster. This is now configurable through normal CLI in Junos 22.4R3-S6:

set system internet-options ip-reassembly-timeout 10
set system internet-options ip6-reassembly-timeout 10

For BNGs with many subscribers and bursty control-plane traffic, this is the recommended mitigation. The value is deliberately below the ISIS hold time, so a stalled fragment is discarded and its buffer returned to the pool before it can block the next hello.

Validating the Fix

After the change, watch the same pair of counters that proved the problem in the first place, along with the adjacency uptime. If the drop is gone, the PFE and RE counters track each other and adjacency uptime climbs monotonically past the previous failure interval:

show isis adjacency extensive | match "Up time"
show isis statistics | match "Total"
show system statistics ip | match reassembly
show log messages | match RPD_ISIS_ADJDOWN
monitor traffic interface ae31.723 no-resolve size 2000

The last command is the one to keep in your back pocket during any future dispute: capturing at 2000 bytes on the sub-interface proves empirically whether hello packets are padded past 1500 bytes, without needing access to the peer's configuration.

Lessons for Service Provider Operators

• Age-out, not interface down, means hello loss — always suspect the control plane before the wire.
• Compare a PFE-side counter against an RE-side counter; the delta localises an internal drop.
• Hello padding is a feature that protects MTU consistency and behaves as a multi-vendor landmine.
• Fragmentation paths are load-sensitive: they only fail when the queue is busy, which is exactly when the network can least afford an outage.
• A lower IP reassembly timeout is a safe, targeted mitigation for control-plane-heavy edge routers.

For more Junos operational topics, see our guides on identifying flapped interfaces on Junos EX, Junos LAN reachability troubleshooting, the Junos troubleshooting command reference, and the multi-vendor CLI cheat sheet. If the peer in your network runs IOS, our Cisco IOS IS-IS configuration guide shows where the default hello-padding behaviour originates; the jumbo frame MTU mismatch troubleshooting guide covers the DF-bit method for finding the real path MTU.

原文链接：https://oasis-tech.net/networks/how-troubleshooting-of-routing-flaps-ends-with-a-new-junos-command