ArubaOS-CX VSF Split Detection: MAD and Split-Brain Recovery

A Virtual Switching Framework (VSF) stack is one logical switch until a VSF link fails in a way that divides the stack into two fragments. Without split detection both fragments believe they are the commander, both keep their ports up, and every dual-homed host suddenly sees the same MAC address, gateway IP and LACP partner on two different devices. This article covers how VSF split detection (multi-active detection, MAD) works on ArubaOS-CX, the two ways to configure it depending on platform, and the exact recovery sequence.

VSF in one page: one control plane, two fragments waiting to happen

VSF turns a set of ArubaOS-CX switches into a single logical device. The members are numbered, one member becomes the commander, a second becomes the standby, and the rest are plain members. Interfaces are addressed by member number, so port 2/1/1 is port 1 of member 2. The whole stack has one control plane, one forwarding database view that is synchronized between members, and one place to manage it from.

The pieces that matter for split detection are:

• VSF links — each member can carry two VSF links (vsf link 1, vsf link 2), and each link can bundle up to four physical ports. Members are cabled in a ring or in a chain.
• VSF priority — a value from 1 to 255, default 128. The highest priority wins the commander election; ties are broken by member MAC address.
• Member roles — commander, standby, member. On a split, the fragment that keeps a valid detection path keeps its role; the other fragment is pulled out of service.

Everything about a stack is fine as long as the VSF links stay up. The interesting failure is not the loss of a single member — it is the loss of enough VSF bandwidth that the members see each other as two independent stacks.

What a split actually looks like

switch# show vsf
switch# show vsf detail
switch# show vsf member
switch# show vsf link
switch# show vsf link detail
switch# show vsf topology
switch# show events -c vsf

show vsf link shows each VSF link and its member ports. A complete loss of one side — for example both 2/23 and 2/24 — removes the whole VSF link, not just 50 % of it. When that happens, check the topology with show vsf topology and the member state with show vsf member, then decide which fragment is the survivor:

• During a split, the fragment that keeps the active management link (or the MAD VLAN connectivity) continues forwarding.
• The losing fragment has its member ports disabled. That is the mitigation, not a bug: it removes the duplicate identity from the network.

On the wire, the symptoms of a split without MAD are unmistakable once you know what to look for. A dual-homed server running LACP receives LACPDUs carrying the same system MAC and system priority from two different sets of ports, so it sees one partner talking twice. ARP replies for the default gateway arrive from two chassis. MAC address tables on downstream access switches show the same MAC alternating between two uplinks. If a first-hop redundancy protocol such as VRRP or active gateway is configured, the virtual MAC and virtual IP exist in two forwarding planes at once. The result is not a clean outage but a slow, ugly degradation: intermittent packet loss, MAC flaps, duplicated broadcasts, and monitoring tools that report contradictory states for the same "device".

The danger grows with the number of dual-homed hosts. Every server, hypervisor or downstream switch that terminates a LAG on the stack has two live paths to a duplicated identity. Traffic that arrives on fragment A and needs to leave on a port belonging to fragment B has no inter-chassis link to traverse, because the split removed the very links that used to carry it.

Two ways to detect a split

AOS-CX gives you two split-detection mechanisms:

• Management-interface based (vsf split-detect mgmt) — the fragment that keeps the shared management path active stays up; the other fragment shuts down. This is the default recommendation on platforms with dedicated OOBM ports.
• MAD VLAN based (vsf split-detect vlan) — front-plane ports are reserved into a MAD VLAN and used as the detection path. This is required on 4100i and 6100 switches, which have no dedicated OOBM interface for split detection.

Only one mechanism is configured at a time, and whichever you choose, the detection path must be independent of the VSF links: if the MAD path shares a cable, a patch panel or a power feed with the VSF links, a single fault can take out both the stack and the mechanism that is supposed to protect it.

Option 1: management-interface based MAD

switch(config)# vsf split-detect mgmt

# Verify
switch# show vsf
switch# show vsf split-detection statistics
switch# clear vsf split-detection statistics

Requirements for the management option to work reliably:

• The management port of every member must be connected and up. A member whose OOBM port is patched but administratively down cannot participate in the detection.
• All member management ports must sit in the same Layer 2 domain so that the fragments can still reach each other after the VSF links fail. If the management network is routed per-site or per-rack, the fragments lose visibility of each other and both stay active.
• The management path must not traverse the stack itself. An OOBM switch that hangs off a downstream port of the VSF stack is not an out-of-band path.

On platforms with dedicated OOBM ports, this is a one-line, non-disruptive change that can be added to a running stack. Because it does not consume front-panel ports or a VLAN, it is also the cleanest option in dense deployments.

Option 2: MAD VLAN based split detection

Where no dedicated management interface exists, or where the management network cannot provide a common L2 segment across members, reserve front-panel ports into a MAD VLAN:

switch(config)# vlan 4000
switch(config-vlan-4000)# exit
switch(config)# interface 1/1/24,2/1/24
switch(config-if-<1/1/24,2/1/24>)# no routing
switch(config-if-<1/1/24,2/1/24>)# vlan access 4000
switch(config-if-<1/1/24,2/1/24>)# no shutdown
switch(config-if-<1/1/24,2/1/24>)# exit

# Point the stack at the MAD VLAN
switch(config)# vsf split-detect vlan 4000

# Verify
switch# show vsf split-detection statistics

Design points for a MAD VLAN:

• Use a VLAN that carries nothing else. The MAD VLAN should be a dedicated Layer 2 broadcast domain, not a repurposed user or transit VLAN.
• Include at least one port per member in the MAD VLAN, and cable those ports to each other directly or through a switch that is not part of the VSF stack.
• Keep MAD ports physically diverse from the VSF links: different line cards, different cable trays, different power domains if possible.
• Document the ports. A MAD port is removed from normal service, and it is easy to mistake a MAC-less access port for a free one during an upgrade.
• On 4100i and 6100 switches the MAD VLAN is the only supported mechanism, so plan the port budget before the stack is cabled, not after.

When split detection is configured, the stack always knows which fragment owns the valid detection path. When it is not configured, both fragments remain active and both keep talking to management platforms — HPE Aruba Networking Central will show two devices where you expect one, and the network will behave accordingly.

Confirming a split and reading the evidence

Split detection is silent until it fires, so the first thing to check during an unexplained outage is the detection counters and the event log:

switch# show vsf
switch# show vsf member
switch# show vsf link detail
switch# show vsf split-detection statistics
switch# show events -c vsf
switch# show interface brief

On the fragment that lost the detection path, member ports show as down in show interface brief while the management interfaces and the MAD VLAN ports stay up. That asymmetry is the signature: the links did not fail, they were disabled by the split-detection action. On the surviving fragment, show vsf still shows the full topology view it had before the split, and the VSF links to the missing member are down.

If the counters in show vsf split-detection statistics have incremented, a detection event has already happened at least once, even if the stack looks healthy now. That usually means the link flapped and recovered on its own — and that a marginal cable, optic or SFP+ cage is still in place waiting to fail again.

Recovery sequence

1. Capture the state before touching hardware. Save show running-config, show vsf, show vsf link detail, show events -c vsf and the split-detection counters. If a TAC case is likely, collect the full support bundle while the split is still visible.
2. Fix the physical cause. Reseat or replace the VSF DAC/AOC cable, replace the optic, or replace the failed member. Check both ends of every VSF link, and remember that a chain topology has a single point of failure on each intermediate member.
3. Watch the fragments merge. The member with the higher VSF priority rejoins as commander, which triggers a commander re-election — expect a short service interruption and a reboot of the standby member that believed it was commander. Traffic on the surviving fragment continues, but its forwarding state is rebuilt as the remote member returns.
4. Verify convergence. show vsf shows one stack with the expected member numbering, show vsf link detail shows all links up and passing traffic, and show vsf split-detection statistics counters stop incrementing.
5. Confirm the control plane is single again. Check that management platforms show one device, that the LACP partner view on dual-homed servers shows a single system ID, and that MAC address tables no longer alternate between uplinks.
6. Clear the counters with clear vsf split-detection statistics so the next split event is visible, and confirm the VSF image auto-sync completed across members so that a future replacement member boots the same firmware as the rest of the stack.

Do not "fix" a split by disabling split detection. Removing the mechanism makes both fragments forward again, which converts a controlled outage into a duplicated-identity event that is much harder to diagnose and can corrupt forwarding tables on downstream devices.

Design rules that prevent the incident

Use a ring topology rather than a chain so a single link failure never creates two fragments in the first place, always configure at least one split-detection mechanism, and reserve MAD/keepalive paths on ports that are physically diverse from the VSF links. Give each member two VSF links where the platform allows it, and spread the member ports of each link across different hardware where possible.

Operationally, treat the split-detection path as production infrastructure: monitor it, include it in change control, and test it during maintenance windows by failing a VSF link on purpose in a lab stack. Keep firmware aligned across members, because a replacement member that boots an older image will auto-sync after joining but will not do so before.

If the stack is already in service, adding MAD is a non-disruptive configuration change — adding ring cabling is not, so plan a maintenance window. On platforms that support both mechanisms, the management-based option is preferred because it consumes no front-panel ports and survives the loss of any single data port.

VSF versus VSX: two different split behaviors

VSF and VSX solve similar problems with very different failure semantics. A VSF stack has one control plane, so a split leaves two fragments fighting over the same identity and the losing fragment must be shut down. VSX keeps two independent control planes connected by an inter-switch link plus a keepalive path, so a lost keepalive does not create a duplicate identity — it triggers the secondary peer to disable its VSX links and leave the primary alone. If your environment runs both, keep the recovery runbooks separate: the commands, the expected states and the blast radius all differ. See our VSX split-brain and keepalive recovery guide for the VSX side of the story.

Quick reference: symptom to first check

• Two devices in Central where one is expected — check whether split detection is configured at all, then look for a VSF link that is down.
• Half the stack's ports down but links show connected — this is the split-detection action on the losing fragment, not a cabling fault.
• Duplicate LACP system ID on a server — a split is in progress or detection is missing; capture show vsf on both fragments.
• Intermittent MAC flaps downstream — verify the MAD path's Layer 2 domain; a routed or partitioned management network will not detect the split.
• Detection counters incrementing with no visible outage — a marginal VSF link is flapping; replace the cable or optic before it fails properly.

Related reading: ArubaOS-CX VSF stacking configuration, VSX configuration guide, NAE scripts and agents for stack monitoring and access versus trunk ports on AOS-CX.

原文链接：https://arubanetworking.hpe.com/techdocs/AOS-CX/10.16/PDF/vsf.pdf