ArubaOS-CX VSX Split-Brain: ISL, Keepalive, Recovery

VSX on ArubaOS-CX is one of the cleaner dual-control-plane designs: two switches keep independent control planes, synchronise state over the Inter-Switch Link, and detect peer death over an out-of-band keepalive path. The failure everybody asks about in interviews and incident reviews is the split brain - both the ISL and the keepalive are down, and each switch believes it is the surviving member. This guide walks the failure sequence, the exact show vsx outputs that tell you where you are, and the recovery levers built into AOS-CX.

Why Both Paths Matter

• ISL (Inter-Switch Link): the data and synchronisation path between the two VSX peers - typically a LAG of 40G/100G links. Down ISL alone does not break VSX; the secondary keeps forwarding on its own until the link returns, and the primary keeps the shared VSX LAGs up.
• Keepalive: a small out-of-band path used to prove the peer is alive when the ISL is gone. Without keepalive, an ISL failure is indistinguishable from a peer failure, and that is exactly when both boxes can start acting primary.

That summary is the one most engineers carry around, and it is close enough to reason about the feature. It is not quite what the hardware does, though, and the difference starts to matter the moment you have to explain an outage to somebody else. The Accuracy note sections below document the precise AOS-CX behaviour.

The Split-Brain Sequence

1. ISL fails (fibre cut, module reset, accidental shutdown of the LAG). VSX holds; keepalive still proves the peer exists, so no role change occurs.
2. Keepalive also fails (wrong VRF, management switch reload, ACL blocking the heartbeat). Neither switch can see the other.
3. Split brain: both members bring up their VSX LAGs and the network has two active gateways, duplicate MAC sources and potential loops. When the ISL is restored there is no automatic reboot of the secondary - the documentation is explicit about this, so recovery must be deliberate.

What the Inter-Switch Link Actually Carries

Treating the ISL as "just a cable between two switches" is what makes split brain feel mysterious. A VSX ISL carries three distinct classes of traffic, and knowing which class broke is most of the diagnosis:

• VSX management and control packets - the peer-distribution protocol that keeps the two control planes coherent. show vsx status inter-switch-link separates these from user traffic, printing Hello Packets Tx/Rx, Mgmt Packets Tx/Rx and Mgmt Packet Drops as independent counters. A climbing Mgmt Packet Drops while the hello counters still look healthy means the control channel is losing information even though the link reports up.
• Synchronised state - MAC address entries, ARP and IPv6 neighbour entries, and whatever features you opted into with vsx-sync: global STP state, VRRP and active-gateway state, QoS and policy globals, static routes, DHCP snooping bindings and more.
• Data-plane traffic - user traffic for VSX LAGs whose member ports live on only one chassis, plus traffic for orphan devices single-homed to the secondary.

Because the third class exists, an ISL failure is never a pure control-plane event: it silently removes capacity from every LAG that spans the pair. Use show running-config vsx-sync and show vsx config-consistency to see exactly which features the pair is supposed to share. Anything not listed there is allowed to differ between the two switches - and after a split, it will.

How Keepalive Reaches Its Verdict

Keepalive is a UDP probe on port 7678 by default (keepalive udp-port changes it). It is sourced from any layer 3 interface - a routed port, a loopback or an SVI - and the path may be direct or routed, as long as it never crosses the ISL. Two timers govern how quickly a peer is declared dead: keepalive hello-interval and keepalive dead-interval. Both default to deliberately conservative values, because a false "peer is dead" is far more destructive than a slow detection.

switch# show vsx status keepalive
Keepalive State           : Keepalive-Established
Last Established          : Thu Jun  8 09:03:01 2018
Last Failed               : Thu Jun  8 09:04:02 2018
Peer System Id            : 58:1f:cf:af:a0:84
Peer Device Role          : primary

Keepalive Counters
Keepalive Packets Tx      : 322
Keepalive Packets Rx      : 121
Keepalive Timeouts        : 0
Keepalive Packets Dropped : 14

Four fields carry most of the diagnostic value. Keepalive State is the verdict. Last Established and Last Failed give you the timeline, which tells you whether the split you are investigating is the first this month or the fifth. Keepalive Packets Dropped is the early-warning counter: it increments when the heartbeat leaves the switch but is policed, queued behind something else, or lost somewhere in the out-of-band path. A keepalive that is nominally "up" while dropping a tenth of its probes will survive steady state and fail exactly when the network is busy - which is precisely when you need it to work.

The Failure Matrix: What Actually Happens

Rather than reasoning from first principles every time, memorise the four states. This is the documented behaviour and the table to quote in a design review:

Accuracy note on the ISL-only case. It is commonly said that "an ISL failure changes nothing because keepalive keeps the pair honest". The documented behaviour is stronger and more interesting: when the ISL goes out of sync while keepalive is still established, the secondary brings its VSX LAG members down. The pair deliberately gives up half the LAG's bandwidth rather than forward with stale MAC and ARP state. The primary keeps all of its members, so the server sees a working - if thinner - LAG throughout.

What Changes During a Split

Once both switches believe the peer is gone, a specific set of things goes wrong simultaneously. Knowing the list lets you confirm a split from the outside, without console access to both boxes:

• Both chassis forward. With split recovery enabled - the default - the secondary restores its VSX LAG member ports after roughly ten missed keepalives, about ten seconds after keepalive itself went down.
• Gratuitous ARP and MAC duplication. Both sides advertise the same virtual MAC addresses and the same anycast gateway, so upstream devices see duplicate MAC entries on two ports and relearn them continuously.
• Active-gateway and VRRP dual-master. An active-gateway VIP is answered by both chassis, and VRRP peers that cannot exchange hellos over the ISL may both claim master.
• MAC and ARP tables diverge. Each side keeps only what it learned locally, so return traffic can land on the wrong chassis and be dropped - the classic "everything is up but half the flows are blackholed" symptom.
• Spanning tree sees two roots. If the pair runs STP, the split looks like two independent topologies and blocked ports can unblock.

The show Commands That Locate the Fault

show vsx brief
show vsx status
show vsx status config-sync
show vsx status peering
show vsx status keepalive
show vsx status shutdown-on-split
show running-config vsx

Read them in this order: keepalive tells you whether the heartbeat is established, peering tells you whether the ISL is up, config-sync shows whether the peers are in sync, and shutdown-on-split shows whether the split-protection policy is armed.

A second set answers the questions those commands raise. show vsx configuration prints the ISL port, hello interval, dead interval, hold time, peer-detect interval, system MAC, device role and the multichassis LAG list on one screen. show vsx status vsx-peer runs the status command on the far side and displays the result locally - the fastest way to see both halves of the picture from a single session. show vsx config-consistency reports features configured on one peer but not the other, and show vsx status linkup-delay lists exactly which interfaces are being held down by the delay timer or by shutdown-on-split.

switch# show vsx status
VSX Operational State
---------------------
ISL channel        : In-Sync
ISL mgmt channel   : operational
Config Sync Status : in-sync
NAE                : peer_reachable
HTTPS Server       : peer_reachable

Attribute          Local              Peer
---------          -----              ----
ISL link           1/1/43             1/1/43
ISL version        2                  2
System MAC         48:0f:cf:af:70:84  48:0f:cf:af:c2:84
Device Role        primary            secondary

Two lines do the heavy lifting. ISL channel is either In-Sync or out of sync; the latter is what you see during and after a split. Config Sync Status tells you whether the peer is carrying a different configuration, which is the fingerprint of a peer that was reconfigured while the ISL was down.

Protection and Recovery

vsx
  inter-switch-link lag 256
  role primary
  keepalive peer 192.0.2.2 source 192.0.2.1 vrf mgmt
  split-recovery
  linkup-delay-timer 120

• split-recovery makes the secondary automatically disable its VSX links after a split, preventing two active primaries from fighting.
• linkup-delay-timer delays ISL-up forwarding so the fabric converges before traffic is admitted.
• Keep the keepalive in a separate failure domain from the ISL - a dedicated management VRF, ideally a different switch and power feed.

Accuracy note on split-recovery. Split recovery mode is enabled by default, and its purpose is the opposite of what the name suggests to most readers. The documented sequence runs like this: when the ISL goes out of sync while keepalive is still established, the secondary VSX LAGs are brought down. If keepalive then also fails - the genuine split condition - the enabled split recovery mode restores the secondary's VSX LAGs, so they come back up after approximately ten missed keepalives. Disabling the mode does the reverse: the secondary keeps its VSX LAGs down, preserving a single-active data path at the cost of capacity. In other words, split-recovery is a policy choice between "use the full bandwidth and accept duplicate forwarding" and "stay down and be safe".

Pair it with vsx shutdown-on-split, configured per interface. Applied to a non-VSX port, LAG or SVI, it shuts that interface down on the secondary during a split - the clean way to keep orphan devices and single-homed uplinks from becoming a loop or a black hole while the pair is confused.

switch(config)# interface lag 5
switch(config-lag-if)# vsx shutdown-on-split
switch(config)# interface vlan 20
switch(config-if-vlan)# vsx shutdown-on-split

Recovering a Split Pair Step by Step

1. Confirm the split before you touch anything. On both switches run show vsx status and show vsx status inter-switch-link. Look for an out-of-sync ISL channel, an operational-but-not-in-sync management channel, and a keepalive whose Last Failed timestamp matches the incident window.
2. Fix the physical cause first. Restore the ISL LAG - check member ports with show interface, show lacp aggregates and the transceiver, and confirm the ports were not administratively shut by an earlier change.
3. Watch the ISL come back into sync. show vsx status inter-switch-link should return to In-Sync and the hello and management counters should start incrementing again. Expect a burst of MAC and ARP synchronisation over the ISL as the pair reconciles the tables it built independently.
4. Do not reload a member reflexively. AOS-CX has no automatic reboot of the secondary after a split, and a reboot is usually unnecessary. Intervene only if the pair will not reach sync and config-consistency checks keep failing.
5. Check the data path, not just the control plane. Verify VSX LAG member counts on both sides, confirm the secondary's LAG members came back, and look for duplicate-MAC complaints on the upstream switches.
6. Let the split-protection state clear properly. Interfaces held by shutdown-on-split or by the link-up delay timer should come back once the ISL is in sync and the delay expires. If they do not, re-apply the interface configuration rather than weakening the policy.

Design Rules That Prevent Split Brain

• Put keepalive in a different failure domain. A dedicated layer 3 link, in a dedicated management VRF, is the documented best practice. The keepalive packet must never be transported over the ISL.
• On 6400 and 8400 series, separate the line cards. The VSX guide is explicit that keepalive and ISL should not share a line card on those platforms, because a single line-card failure then removes both paths at once - producing an instant split brain.
• Enable configuration sync deliberately. Use vsx-sync per feature, and config-sync disable only where you have a specific reason. Two peers with silently different configurations disagree the moment the ISL drops.
• Size the link-up delay to your control plane. The delay timer exists so MAC and ARP synchronisation completes before LAG members are admitted. Too short, and servers see a working LAG in front of an empty forwarding table.
• Protect the edges. vsx shutdown-on-split on orphan ports and single-homed uplinks removes the most common source of loops during a split.

Common Misconfigurations and Their Symptoms

• Keepalive configured in the wrong VRF. The switches have layer 3 reachability in the default VRF, but the keepalive sources from a management VRF with no route back. Symptom: Keepalive State flapping around every management change, and a real split the first time the ISL fails.
• Keepalive reachable only through the ISL. The most dangerous of all: both paths look healthy in steady state, so nothing appears wrong - until the ISL fails and the keepalive fails with it, producing an instant split.
• Undersized dead-interval on a slow path. A keepalive over a congested management network needs a dead-interval long enough to absorb normal jitter, not the shortest value the CLI accepts.
• Policy configured on one peer only. Split recovery and shutdown-on-split need to agree across the pair; the secondary's behaviour is what actually decides the data path.
• Assuming VSX makes spanning tree unnecessary. VSX removes the need to block ports between the pair, but edge loops, orphan-port loops and mis-cabled access switches still exist - and they become visible during the split.

Post-Incident Checklist

1. Record show vsx status from both peers after restoration and confirm "in sync".
2. Verify VSX LAG membership and that no upstream device sees duplicate MACs.
3. Review show logging -r for keepalive and ISL flaps, then fix the root cause (usually the out-of-band path, not the data plane).
4. Compare show vsx config-consistency on both peers and reconcile anything the split allowed to drift.
5. Capture the keepalive counters - Tx, Rx, Timeouts and Dropped - as a baseline. A slowly rising Dropped count is what turns the next ISL failure into a second outage.
6. Confirm the link-up delay timer expired, and that delay-held and shutdown-on-split interfaces are back up.

Frequently Asked Questions

Does the secondary switch reboot after a split? No. AOS-CX does not automatically reload a member after a split; the pair reconciles once the ISL is back in sync.

Can I run VSX without keepalive? Yes, and the pair forwards happily - with split detection disabled. That is the configuration behind most "we had a VSX pair and both switches became active" incidents.

How is VSX split brain different from VSF split brain? VSF is a single control plane with one elected commander, and split detection uses MAD. VSX is two control planes with equal roles, and split detection is the keepalive plus the failure matrix above - different mechanism, different recovery procedure, different commands.

Related reading: ArubaOS-CX VSX configuration guide, VSF split detection (MAD) and split-brain recovery, ArubaOS-CX access vs trunk port comparison and ArubaOS-CX VRRP and active gateway.

原文链接：HPE AOS-CX VSX Guide: Split brain scenario