Juniper MC-LAG: ICL vs ICCP Failure Behavior Compared

MC-LAG runs two independent control paths on top of the same pair of chassis: ICCP for control-plane synchronisation and the ICL for the cross-chassis data path. Because the two fail independently, the same "MC-LAG is broken" symptom can mean two completely different forwarding states. This article puts the two failure modes side by side: what each peer does with its LACP bundle members, when traffic is blackholed instead of looped, and which show commands tell you which of the two paths actually broke. The commands below were validated against QFX and MX MC-LAG deployments described in Juniper's feature documentation and public field reports.

What ICCP and the ICL each do

ICCP (Inter-Chassis Control Protocol) is the control channel. It runs over an IP-routed link — typically an out-of-band management network or a dedicated L3 interface — and synchronises the LACP state machine, the shared LACP system ID and MAC address, ARP/ND entries, IGMP/PIM state and the active/standby role for each MC-LAG bundle.

The ICL (Inter-Chassis Link) is the data path. It is an aggregated Ethernet bundle between the two members that carries cross-chassis traffic: when the downstream device hashes a flow to member A but member A's link to the destination sits behind member B, the frame is forwarded over the ICL rather than dropped.

That split matters because a single member can still forward correctly with only one of the two paths alive — and can be a loop source with neither.

Failure matrix: ICL down vs ICCP down

ICCP failure: backup liveness detection decides everything

ICCP uses a keepalive mechanism, but when the ICCP session itself goes operationally down the peers cannot rely on it to prove the far side is alive. This is exactly what backup liveness detection solves: each peer sends liveness requests to a backup peer IP — normally the same address used for the ICL or a management path — and if the peer does not answer inside the configured interval the peer is declared down.

set protocols iccp local-ip-addr 169.254.0.1
set protocols iccp peer 169.254.0.2 session-establishment-hold-time 50
set protocols iccp peer 169.254.0.2 backup-liveness-detection backup-peer-ip 172.16.214.18
set protocols iccp peer 169.254.0.2 liveness-detection minimum-interval 150
set protocols iccp peer 169.254.0.2 liveness-detection multiplier 3

Once the peer is declared down, the per-bundle events statements decide what happens to forwarding. Two knobs do the heavy lifting:

• force-icl-down — when ICCP to the peer is lost, tear down the ICL so that no traffic can be sent into a peer whose state is unknown.
• prefer-status-control-active — keep the locally configured active member active even when it cannot reach the peer.

set interfaces ae2 aggregated-ether-options mc-ae mc-ae-id 2
set interfaces ae2 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae2 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae2 aggregated-ether-options mc-ae mode active-active
set interfaces ae2 aggregated-ether-options mc-ae status-control active
set interfaces ae2 aggregated-ether-options mc-ae events iccp-peer-down force-icl-down
set interfaces ae2 aggregated-ether-options mc-ae events iccp-peer-down prefer-status-control-active

Leaving prefer-status-control-active unset and using status-control standby on one member is the safer default: it gives you a deterministic standby that will not fight for the active role when ICCP drops.

ICL failure: the containment mechanism

When the ICL goes down but ICCP stays up, the peer that detects the data-path loss keeps its MC-LAG links in standby, so the downstream device drains that member and keeps forwarding through the survivor. This is a controlled re-route, not an outage — provided the surviving member has enough bandwidth for the aggregated load. Sizing the MC-LAG so that either member can absorb the full downstream bundle is therefore a design requirement, not a nicety.

Watch the init-delay-time value as well. It delays bringing MC-LAG links up after a rebuild to give ICL and ICCP time to converge, which prevents a short window where the first member comes up before the ICL is forwarding and blackholes cross-chassis traffic.

Verification commands that separate the two cases

# Is ICCP up, and is liveness detection happy?
show iccp
show iccp log

# Which member is active / standby, and what is the ICL state?
show interfaces ae2 mc-ae
show mc-ae status

# Is the LACP system ID identical on both members?
show lacp interfaces ae2
show lacp interfaces ae2 extensive | match "System"

# Confirm the ICL bundle carries traffic and has no errors
show interfaces ae1 extensive | match "error|CRC"
monitor interface ae1

If show iccp reports the session down and the LACP system MAC still matches, you are in the liveness-detection branch: check the backup peer IP for reachability before touching the LACP configuration. If the ICCP session is up but one member shows the bundle as standby while the ICL shows down, you are in the ICL branch — chase optics, LAG member state and the ICL VLAN trunk rather than ICCP.

Recovery runbook

1. Confirm the failing path with the commands above; never reset both peers at once.
2. For ICCP loss, verify the routed path and the backup peer IP first, then re-establish the session and confirm both members show identical LACP system IDs.
3. For ICL loss, repair the bundle members, then clear LACP counters and confirm MAC address learning stabilises.
4. After any event, check for stale ARP entries on the IRB: MC-LAG keeps the peer's ARP table in sync via ICCP, and a partial sync leaves hosts unreachable until entries age out.

Treat ICCP and ICL as two separate monitored services. Alerting on both — plus on LACP system ID mismatch between members — catches the split-brain case before hosts start reporting duplicate IP warnings.

Timer and detection tuning: the numbers behind the behaviour

The failure behaviour above is only as good as the timers that detect it. Three sets of values determine how fast each peer learns that something is wrong, and they must be consistent on both chassis.

Reading the multiplier correctly matters: with a 150 ms minimum-interval and a multiplier of 3, the peer is declared down after roughly 450 ms of unanswered liveness requests. That is fast enough to be useful and slow enough to survive a control-plane stall, which is why it is a common production value.

Keep init-delay-time in mind as an availability knob rather than a detection knob. It exists to prevent a race: after a member rebuilds, the MC-LAG bundle is administratively up before the ICL can forward cross-chassis traffic. Without the delay, the first few hundred milliseconds of traffic can be blackholed for flows that need the ICL. With it, recovery takes a predictable couple of minutes but is clean.

Monitoring the two paths separately

The single most valuable operational change for an MC-LAG deployment is separate alerting on ICCP and on the ICL, because the bundle state stays up in both failure branches and therefore tells you nothing.

# Poll these three values and alert on each independently
show iccp | match "Session"
show interfaces ae2 mc-ae | match "State|ICCP"
show lacp interfaces ae2 | match "System ID"

• Alert if any MC-LAG member pair reports different LACP system IDs — that is the split-brain leading indicator.
• Alert if ICCP has been down for more than two polling intervals.
• Alert on ICL error counters (CRC, input errors) even while the bundle is up, since a degrading ICL is the mode that eventually triggers a data-path failure during a maintenance window.
• Log a change event whenever a member transitions between active and standby; repeated transitions are a symptom of unstable timers or a flapping link.

Finally, keep a documented recovery order for the worst case: ICCP down and ICL down. Bring up the routed path that carries ICCP first, verify the session re-establishes and the system IDs match, and only then allow the ICL to come back. Doing it in the opposite order risks a hard loop while both members believe they are alone and active.

Internal references

For the step-by-step failure-scenario walkthrough with the LACP system ID change matrix, see Juniper MC-LAG ICCP failure scenarios and LACP system ID behaviour. The protocol-level view of ICCP keepalives, BFD and LACP system ID is covered in the MC-LAG ICCP, BFD and LACP system ID deep dive, and the split-brain configuration details live in Junos MC-LAG backup liveness detection and split-brain configuration.

原文链接：https://apps.juniper.net/feature-explorer/feature/2477