Juniper MC-LAG Design: ICCP Liveness, ICL and Hold Timers

MC-LAG lets two switches present one LACP bundle to a downstream server or switch, removing the chassis as a single point of failure - but the design details decide whether failures converge cleanly or cause flaps. The two control/data paths have different jobs: ICCP (Inter-Chassis Control Protocol) synchronizes state between the peers, while the ICL (inter-chassis link) forwards data traffic between them. When engineers tune one without the other, a link failure that should be invisible turns into a double failover. This article covers the design parameters that matter: ICCP transport and liveness, ICL sizing, and hold timers.

It is worth being precise about what MC-LAG does and does not do. It is a layer-2 multihoming technique: the two peers share a common LACP system ID and a common system priority so that the downstream device believes it is talking to one switch, and only one peer - the one elected active for that MC-LAG interface - forwards traffic for a given downstream link. The peers keep each other informed over ICCP, and any frame that arrives on the standby peer and must leave through a link owned by the active peer is relayed across the ICL. MC-LAG does not distribute routing state, does not synchronise MAC tables on its own, and does not make the peers independent; it makes them a tightly coupled pair whose coupling is what makes the failure behaviour deterministic.

ICCP: Control over TCP, Failure Detection over BFD

MC-LAG peers exchange control information over ICCP, which runs over TCP/IP and therefore needs reachability between the two devices. The Junos configuration anchors ICCP on the loopback addresses and uses BFD for fast failure detection:

set protocols iccp local-ip-address 10.255.0.1
set protocols iccp peer 10.255.0.2 session-establishment-hold-time 50
set protocols iccp peer 10.255.0.2 redundancy-group-id-list 1
set protocols iccp peer 10.255.0.2 liveness-detection minimum-interval 60
set protocols iccp peer 10.255.0.2 liveness-detection multiplier 3

Three design points hide in these lines:

• Loopback as the ICCP source - a dedicated link between the peers is not strictly required for ICCP (it is for the ICL data path), but using loopback addresses makes the ICCP session independent of any single inter-chassis link, and BFD on top gives ~180 ms failure detection with a 60 ms interval and multiplier 3.
• session-establishment-hold-time 50 - recommended so the ICCP session comes up quickly and the redundancy group stabilizes early.
• redundancy-group-id-list must match on both peers and be referenced by the MC-AE configuration - mismatches here produce MC-LAG interfaces that never activate.

The transport question deserves more attention than it usually gets. ICCP is a TCP session on port 2020 by default, riding on whatever IGP path exists between the two loopbacks. If that path traverses the ICL itself, then an ICL failure takes down both the data relay and the liveness path at the same moment, and the two peers discover the failure by losing each other entirely rather than by seeing a clean link event. The better design gives ICCP a path that does not depend on the ICL: either a pair of direct inter-chassis links with an IGP running across them and the ICCP session pinned to a loopback reachable over both, or a separate management/control network. The goal is that ICCP liveness and ICL availability fail independently, so the hold-timer rule below has a chance to do its job.

BFD on ICCP is not optional in production, because TCP alone detects a dead peer only when the socket times out, which can be tens of seconds. BFD turns that into single-digit-hundred-millisecond detection, and it detects the failure that matters most: a peer that is still reachable at layer 3 but has stopped processing control traffic. On MX and QFX platforms ICCP liveness detection is signalled by BFD, and the timers are negotiated - which means a mismatch between the two peers' configurations can leave you with a session that works but detects failures at the slower of the two rates. Configure both peers identically and verify the negotiated values rather than assuming them:

show iccp
show iccp peer 10.255.0.2
show bfd session extensive | find "iccp"
show redundancy-group

The ICL and the Hold-Time Rule

The ICL forwards data traffic between the peers (for example when a downstream server sends to a MAC learned only on the peer). Because ICCP and ICL are separate paths, a failure can hit them in either order - and the order decides whether the MC-LAG flaps:

set interfaces ae0 aggregated-ether-options hold-time up 1000 down 1000

Configure a hold-down timer on the ICL member links that is greater than the BFD timer for the ICCP session. The reasoning: if the ICL is reported down before ICCP detects the peer is gone, the status-control standby node sees its MC-LAG interface flap and convergence is delayed. Holding the ICL down announcement until ICCP/ BFD has already declared the peer dead lets the failure be handled as one deterministic event instead of two cascading ones. This is the single most commonly missed parameter in MC-LAG designs.

Sizing the ICL is the second half of the same problem, and it is a bandwidth calculation rather than a configuration choice. In steady state the ICL carries only the traffic that the standby peer cannot switch locally - typically a minority of flows, plus broadcast and unknown-unicast floods. When the active peer fails, everything the downstream devices were sending to the failed peer's links either shifts to the surviving peer's own links (if the downstream device runs LACP and re-hashes) or is relayed across the ICL. Model the worst case: all downstream bandwidth concentrated on one peer, minus the capacity of that peer's own member links, must fit inside the ICL. A single 40 G ICL between two switches that each aggregate 100 G of downstream access is fine in steady state and a hard bottleneck during a failure, which is precisely when you need it most.

The hold-time rule generalises beyond the ICL. Every timer in the design should be ordered so that a single physical event produces a single state change: physical link detection first, then BFD on ICCP, then ICL hold-down, and only then any LACP or STP reaction on the MC-AE. When two mechanisms react to the same event with similar timers, the result is a race, and races produce the intermittent flaps that are impossible to reproduce in a lab. Document the timer hierarchy in the design document as a table with millisecond values, because six months later nobody remembers which timer was supposed to be longer.

Failure Behavior: ICCP Down vs ICL Down

Knowing what the peers do in each scenario prevents "it worked in the lab" surprises:

• ICCP down: the backup MC-LAG peer changes its LACP system ID and brings down the MC-AE interfaces - the deterministic behavior that prevents the downstream device from seeing two different LACP partners and flapping between them (detailed in our ICCP failure: LACP system ID guide and the QFX-specific follow-up here).
• Double failover ordering: if ICCP goes down and then the active MC-LAG peer goes down, or an MC-LAG peer in active mode is lost, the sequence matters - Junos documentation describes these scenarios explicitly (secondary peer with status control set to standby becoming inactive, and L3 traffic to the standby peer not being redirected when the remote peer's status is unknown).
• Split-brain: if the inter-peer link fails but both peers stay up, both devices may believe they are primary; the ICCP/keepalive design and the LACP system-ID behavior above are what break the tie.

Two further behaviours deserve to be in the runbook because they surprise people during incidents. First, when ICCP is down and the surviving peer has standalone capability configured, the MC-LAG interfaces can be brought up in standalone mode - the downstream device keeps forwarding, at the cost of an active/active pair becoming two independent switches with potentially conflicting MAC or ARP state. That is a deliberate trade of correctness for availability, and it should be a decision someone made on purpose, recorded in the design, with its consequences understood. Second, if the ICCP session is up but a redundancy group is in an unexpected state (show redundancy-group disagreeing between peers), the fault is almost always in the redundancy-group-id-list or the MC-AE configuration rather than the transport, and the fix is a configuration diff between the two peers, not a session clear.

Design Checklist

• ICCP between loopbacks with BFD liveness (60 ms x 3 is a sane starting point).
• ICL sized for worst-case redirected traffic (it carries what the peer cannot switch locally).
• ICL hold timers longer than ICCP BFD detection.
• Identical redundancy-group-id-list and consistent MC-AE config on both peers.
• Understand the ICCP-down failure behavior before deploying - the underlying mechanism is explained in our MC-LAG/ICCP/BFD deep dive.

Verification and Day-Two Operations

A design is only as good as the commands that prove it is still true. Run these after every change and record the output in the change ticket:

show iccp
show iccp statistics
show redundancy-group 1
show lacp interfaces ae0
show interfaces ae0 extensive | match -i "hold-time|redundancy"
show interfaces ae0.0 mc-ae lacp-status
show configuration protocols iccp | display set

Then test the two failures that matter, during a maintenance window, on a non-production pair. Pull one ICL member and watch which side reacts first and how long the MC-AE stays up. Pull the active peer's power and confirm the downstream device re-hashes its LACP bundle within the expected time. Kill the ICCP transport while both peers remain up and confirm that exactly one peer takes over and the LACP system ID changes as designed. If any of those tests produces a flap on the downstream device, the timer hierarchy is wrong and the design is not finished - regardless of what the configuration says.

Monitoring is the other day-two task. The counters that indicate trouble before a user notices are ICCP session state changes (any transition is an event worth an alert), BFD session state changes, ICL interface error counters, and MC-AE interface state. A pair that has been stable for a year and then reports three ICCP transitions in an hour is telling you about a path problem between the loopbacks - and the underlying detection mechanics are covered in Junos BFD liveness detection and MC-LAG integration.

原文链接：https://packet-expert.org/2016/09/15/multi-chassis-lag/ · Juniper TechLibrary: Advanced MC-LAG Concepts