AOS-CX Packet Capture: tcpdump on Aruba CX Switches

Aruba AOS-CX switches are built on a Linux foundation, which means the familiar tcpdump utility is available right on the switch for packet capture — no external taps or port mirrors to a server required. This post shows how to configure a mirror session that sends traffic to the CPU, then run diag utilities tcpdump with granular Linux-style filters. You will also see how captured VXLAN-encapsulated traffic is automatically decoded so you can read the original ICMP payload inside the tunnel.

Why Packet Capture on AOS-CX Is Different

On an ASIC-based switch, most traffic never touches the CPU, so you cannot simply sniff a port like on a server. The critical first step is to configure a mirror session with the CPU as the destination:

Leaf1(config-mirror-1)# show run cur mirror
session 1
 destination cpu
 source interface 1/1/4 both
 source interface 1/1/3 both
 enable

Two properties of the CPU destination are worth internalising before you build the session. First, the mirror has to terminate on the switch's own control plane, which is a shared, rate-limited resource: mirroring a busy 25 GbE uplink to the CPU will drop the vast majority of frames and can starve the protocols that keep the box reachable. Mirror interface-to-interface for heavy traffic, and reserve CPU mirroring for low-rate or control-plane traffic that you need to read directly with tcpdump. Second, the CPU mirror shows the packet exactly as the ASIC handed it up, which is why VXLAN frames appear with the outer header intact and the inner packet decoded — you are watching the real encapsulation, not a synthetic copy.

The session is ordinary configuration, so it survives a reboot if you save it and it can be applied to a VSX pair from any management plane:

Leaf1(config)# mirror session 1
Leaf1(config-mirror-1)# destination cpu
Leaf1(config-mirror-1)# source interface 1/1/4 both
Leaf1(config-mirror-1)# source interface 1/1/3 rx
Leaf1(config-mirror-1)# enable
Leaf1(config-mirror-1)# exit
Leaf1# show mirror

The both keyword mirrors transmit and receive; rx and tx narrow the capture when you already know which direction matters — for example, when you want to see exactly what a server sends before playing with spanning-tree or LACP. Every source interface you add increases the CPU load, so remove sources as soon as the capture is finished.

Running tcpdump from the AOS-CX CLI

The diag command unlocks Linux utilities on the switch. The example below captures VXLAN packets (UDP 4789) in both directions using the command keyword, which accepts traditional Linux tcpdump syntax:

diag utilities tcpdump command -t udp dst port 4789 or udp src port 4789 -A

Breaking the filter apart shows why the shape matters. The expression udp dst port 4789 or udp src port 4789 matches both directions of a VXLAN flow; writing only dst port 4789 would hide the return traffic and make a healthy ping look one-way. The -A flag prints each packet as ASCII, which is ideal for ICMP, HTTP and DHCP payloads because you can read the application data without exporting a pcap. The -t flag removes the timestamp prefix, keeping the scrollback compact when the capture window is short.

Other flags behave exactly as they do on a Linux server, and the ones you will reach for most are:

diag utilities tcpdump command -nn -c 20 icmp
diag utilities tcpdump command -nn host 10.0.113.68 and not port 22
diag utilities tcpdump command -e vlan

-nn stops DNS and service-name resolution, which keeps output fast and avoids name lookups from the switch itself. -c 20 stops after twenty packets so the capture cannot flood your SSH session. -e prints the Ethernet and 802.1Q headers, which is the fastest way to confirm whether a frame arrives tagged on VLAN 10 or untagged on the native VLAN. Combine -e with a host filter when you are chasing a VLAN mismatch between two switches.

Reading VXLAN-Decoded Output

Because the original IP PDU is carried inside the VXLAN header, tcpdump displays both the outer tunnel and the inner packet. In a two-way ping across an L3 VNI you will see the VXLAN frame followed by the decapsulated ICMP echo request and reply:

IP 10.51.100.0.21484 > 10.51.100.6.4789 : VXLAN, flags I, vni 20001
IP 10.0.113.68 > 10.0.113.5 : ICMP echo request, id 10889, seq 1, length 108

This is a fast way to prove which member of a VSX pair actually received a packet, or to verify whether VXLAN traffic is being decapsulated correctly in a spine-leaf fabric. Read the pair of lines as one event: the outer line proves the underlay delivered the encapsulated frame to this VTEP, and the inner line proves the VNI mapping and the tenant MAC lookup resolved to a real host. If you see the outer line but never the inner one, the problem is on the decapsulation side — a missing VNI, an incomplete MAC-IP entry, or a VSX peer that owns the tunnel endpoint.

Three quick interpretations cover most cases:

• Outer only, repeatedly: the VTEP receives the frame but the VNI is not mapped or the inner destination is unknown to this switch.
• Request without reply: the fabric forwards fine, the host or its return path is the problem.
• VNI mismatch between two captures: the two ends of the conversation are not on the same L2 or L3 VNI, a classic after a fabric expansion.

Going Deeper with tshark

The related tshark utility is the CLI version of Wireshark and decodes packets in greater detail. A useful combination is tcpdump for very granular filtering, followed by tshark when you need to inspect a specific flag inside a field — for example, pulling the BGP state machine from a peering session or reading the LACP actor state bits in a bundle that refuses to come up. Because both tools run from the same Linux userland, the workflow is simply: capture with tcpdump to answer "is the packet here at all", then switch to tshark to answer "what exactly is inside it".

Remember to remove or disable the mirror session when the investigation ends:

Leaf1(config)# no mirror session 1
Leaf1# show mirror

Leaving a CPU mirror enabled is one of the more common causes of unexplained high CPU on an otherwise idle switch, and it can also mask the very symptom you were called in to fix.

Practical Capture Workflow

1. Confirm which switch actually holds the flow — use the VSX active-forwarding information first so you mirror on the right member.
2. Build a narrow mirror session: one or two source interfaces, one direction if possible.
3. Filter aggressively with -nn, a host or port expression, and a packet count.
4. Read the output live for ICMP and DHCP, or capture to a file when you need Wireshark's richer dissectors.
5. Disable the session and verify with show mirror.

Choosing Between CPU and Interface Mirror Destinations

Not every investigation belongs on the CPU. When you need to hand a capture to a colleague or feed it to a protocol analyser, mirror the traffic to a spare port and run Wireshark or tshark on a laptop attached to that port; nothing is rate-limited by the switch's control plane, and the pcap is complete. The trade-off is that a mirror-to-port destination must be a dedicated interface — an analyser plugged into a live VLAN receives both the mirrored copy and whatever the VLAN delivers normally, which makes the capture confusing for anyone who does not know the mirror is running.

The rules of thumb that keep the two approaches straight in daily work are simple. Use the CPU destination for control-plane protocols, low-rate flows and anything you want to read in the SSH window with tcpdump's own dissectors. Use an interface destination for bulk traffic, for sustained captures and whenever the resulting file will be opened in Wireshark. And never leave either session enabled: CPU mirrors burn control-plane cycles, and interface mirrors consume a port plus bandwidth for as long as they run.

Both destination types are configured the same way, which makes switching between them a one-line change:

Leaf1(config)# mirror session 2
Leaf1(config-mirror-2)# destination interface 1/1/48
Leaf1(config-mirror-2)# source interface 1/1/10 both
Leaf1(config-mirror-2)# enable
Leaf1(config-mirror-2)# exit
Leaf1# show mirror

Reading DHCP, LACP and ICMP Without a pcap File

Most of what a network engineer needs to prove can be read directly in the terminal, without exporting anything. A DHCP negotiation, for example, is four packets and fits comfortably in a scrollback buffer:

diag utilities tcpdump command -nn -A udp port 67 or udp port 68

The -A flag makes each packet readable, so you can see the requested address, the offered address and the DHCP option list straight away. This is the fastest way to distinguish "the client never sent a request" from "the relay dropped it" from "the server answered and the client ignored it" — three faults that look identical from the host. The same technique works for LACP, where you are looking for the partner's system ID inside the PDU rather than a decoded field, and for ARP, where the sender MAC in the request often reveals a duplicated address long before the ARP table shows anything unusual.

Capture Cost and Performance Limits

A capture is not free. Mirroring to the CPU competes with the protocols that keep the switch manageable, so on a busy leaf it is normal to see tcpdump itself drop packets while the interface copy stays complete. A few practical guidelines keep the impact negligible: filter as early as possible in the expression, keep the capture under a minute or two, use a packet count, and prefer the narrowest source interface that still sees the traffic. If you find yourself mirroring a whole uplink bundle to the CPU to chase a single flow, stop and move the session to an analyser port instead — the CPU is the wrong tool for that job, and the dropped packets will mislead you into chasing a problem that does not exist.

For related campus and data center troubleshooting, see our notes on clearing ArubaOS configuration, MLNX-OS breakout cable link troubleshooting and the SONiC troubleshooting guide. For the filter side of the same workflow, see our tcpdump command examples and filter guide.

原文链接：https://constantpinger.home.blog/2021/11/05/aruba-aos-cx-packet-capture-including-vxlan-decoding/