tcpdump Packet Capture Guide: BPF Filters and Troubleshooting - 夜莺博客

tcpdump Packet Capture Guide: BPF Filters and Troubleshooting

When a server "can't reach the network," the fastest way to separate fact from guesswork is a packet capture — and tcpdump is the tool that works everywhere, from a production server to an embedded Linux appliance. The key to using it well is the BPF filter: tcpdump compiles your expression to kernel bytecode so only the packets you care about are copied to userspace. This guide covers interface selection, the BPF primitives that matter, reading TCP flags, and the practical troubleshooting scenarios where tcpdump shines.

Installation and Interface Discovery

# Debian/Ubuntu
sudo apt install tcpdump
# RHEL/CentOS/Fedora
sudo yum install tcpdump
# List capture interfaces
sudo tcpdump -D
# Capture on all interfaces
sudo tcpdump -i any

The -D output is worth reading carefully before you capture anything, because a modern Linux host has far more interfaces than a physical NIC. A typical KVM/Proxmox or container host lists the loopback device, the management NIC, bridges such as vmbr0, bonds such as bond0, VLAN sub-interfaces such as eth1.100, and veth pairs belonging to containers. Capturing on the wrong one is the single most common reason a "capture shows nothing" report turns out to be false — the traffic left through a different interface.

$ sudo tcpdump -D
1.eth0 [Up, Running, Connected]
2.any (Pseudo-device that captures on all interfaces) [Up, Running]
3.lo [Up, Running, Loopback]
4.nflog (Linux netfilter log (NFLOG) interface)
5.nfqueue (Linux netfilter queue (NFQUEUE) interface)
6.docker0 [Up, Running]
7.veth9f3a1c2 [Up, Running]

On Debian and Ubuntu the installed binary usually carries file capabilities already, so any user in the pcap group — or any user at all, depending on the build — can capture without sudo. If it does not, grant the capabilities rather than making the tool setuid root.

getcap /usr/bin/tcpdump
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/tcpdump
capsh --print | grep cap_net_raw

How the BPF Filter Actually Works

tcpdump does not decide anything in userspace. It hands your expression to libpcap, which compiles it into classic BPF (cBPF) bytecode and attaches it to the capture socket. From that moment the kernel evaluates the program for every packet on the interface and only copies matching frames upward. That is why a precise filter is not merely cosmetic: on a busy 10 GbE link, an unfiltered capture will overflow the ring buffer and report thousands of dropped packets, while 'tcp port 443 and host 10.0.0.5' costs almost nothing.

You can inspect the generated program — useful when a filter behaves unexpectedly:

sudo tcpdump -i eth0 -d 'tcp port 443'        # short disassembly
sudo tcpdump -i eth0 -dd 'host 10.0.0.1'      # C-style array
sudo tcpdump -i eth0 -ddd 'icmp' | head -20   # raw decimal bytecode

Two consequences follow. First, the filter is applied before the snaplen truncation, so a bad filter cannot be "fixed" afterwards — you must capture again. Second, a filter that references protocols the kernel data path cannot see (for example, trying to match inner headers of an encapsulated tunnel) will silently match nothing. When that happens, capture broadly and filter in Wireshark instead.

Interface Selection and Promiscuous Mode

By default tcpdump puts the interface into promiscuous mode so it sees traffic not addressed to the host's own MAC. That is exactly what you want when a server is routing, bridging, or running containers, and exactly what you do not want on a heavily loaded switch uplink where every frame on the segment gets copied. -p disables promiscuous mode.

sudo tcpdump -i eth0 -p -c 5                  # own traffic only
sudo tcpdump -i any -c 10                     # all interfaces, cooked header
sudo ip netns exec blue tcpdump -i veth-blue -nn
sudo tcpdump -i bond0 -nn 'not host 10.0.0.9'

Remember that -i any uses the Linux "cooked" capture header (LINUX_SLL). It sees packets on every interface including loopback and netns veths, which makes it the fastest way to answer "is the traffic leaving at all?" — but it cannot reliably match Ethernet-specific expressions such as ether host, and it reports the interface name in each line rather than a link-layer address.

BPF Filter Primitives

BPF expressions are evaluated in the kernel — packets that do not match are discarded before reaching userspace, minimizing CPU and memory impact.

host 192.168.1.5            # src OR dst = 192.168.1.5
src host 10.0.0.1           # source only
dst host 10.0.0.2           # destination only
net 192.168.1.0/24          # subnet
port 443                    # either direction
src port 443 and dst host 10.0.0.5
vlan 10                     # 802.1Q tagged frames for VLAN 10
ether host aa:bb:cc:dd:ee:ff  # specific MAC
tcp and port 22             # boolean combination
not port 22

A filter is built from three kinds of qualifier. The type says what to match (host, net, port, portrange); the direction says which side (src, dst, src or dst, src and dst); and the protocol says at which layer (ether, ip, ip6, arp, tcp, udp, icmp). If you omit a qualifier, tcpdump applies a sensible default: no direction means src or dst, and no type means host. So bare 10.0.0.5 is shorthand for host 10.0.0.5.

portrange 8000-9000
ip proto 47                     # GRE (protocol number, not a keyword)
proto ospf                      # by name where tcpdump knows it
icmp[icmptype] == icmp-echo     # ICMP echo requests only
tcp[13] & 2 != 0                # raw offset: SYN flag set
ip6 host 2001:db8::53
mpls 100                        # MPLS label 100
ether proto 0x88cc              # LLDP frames
udp port 514                    # syslog
not arp and not port 22
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420   # payload starts with "GET "

The bracket syntax is the escape hatch for anything tcpdump has no keyword for. tcp[13] is the flags byte of the TCP header, icmp[0] is the ICMP type, and arbitrary offsets let you match application payloads — the last example above matches an HTTP request line beginning with GET . Use it sparingly: offset filters are brittle when headers carry options.

Combining Primitives: Operators, Parentheses and Precedence

Boolean operators are and / &&, or / ||, and not / !. The negation binds tighter than concatenation, and and binds tighter than or, so parentheses matter. Always single-quote the whole expression, otherwise the shell eats the parentheses and the pipe characters and you get a confusing "syntax error" from tcpdump instead.

sudo tcpdump -nn -i any 'tcp and (port 80 or port 443) and not host 10.0.0.9'
sudo tcpdump -nn -i any 'host 10.0.0.5 and (tcp port 22 or tcp port 3306)'
sudo tcpdump -nn -i eth0 'not port 22 and not arp and ip'
sudo tcpdump -nn -i eth0 'udp and (port 53 or port 123 or port 5353)'

One classic trap: not port 22 is valid, but not port 22 and port 80 reads as (not port 22) and port 80 for every packet, which is fine — whereas not (port 22 and port 80) is always true. Reach for explicit parentheses whenever you mix negation with conjunctions.

Reading TCP Flags

Flag letters in output: [S] SYN, [S.] SYN-ACK, [.] bare ACK, [P.] PSH-ACK (data), [F.] FIN, [R] RST. A burst of [S] with no [S.] means the remote side is not responding; an immediate [R] after [S] means the port is closed or a firewall is rejecting.

The flags byte can also be named in the filter itself. tcpdump knows tcp-fin, tcp-syn, tcp-rst, tcp-push, tcp-ack, tcp-urg, tcp-ece and tcp-cwr, so you never have to remember bit values.

# SYN packets only (SYN flood detection)
sudo tcpdump -nn -i eth0 'tcp[tcpflags] == tcp-syn'
# RST packets (connection resets)
sudo tcpdump -nn -i eth0 'tcp[tcpflags] & tcp-rst != 0'
# Pure SYN-ACK responses
sudo tcpdump -nn -i eth0 'tcp[tcpflags] == tcp-syn|tcp-ack'
# Any teardown (FIN or RST)
sudo tcpdump -nn -i eth0 'tcp[tcpflags] & (tcp-fin|tcp-rst) != 0'
# Zero-window probes and window updates (ACK with no payload)
sudo tcpdump -nn -i eth0 'tcp[tcpflags] == tcp-ack and tcp[14:2] == 0'

Read a handshake line by line and the diagnosis writes itself: S out, S. back, . out is a healthy connection. S repeated with no answer points at a routing, firewall, or listener problem on the far side. S followed immediately by R means something actively rejected it — a closed port or a REJECT rule. S, S., then R. from the client usually means the application was not listening after all. And a connection that reaches . then repeats the same sequence number with [P.] several times is retransmitting, which is a loss problem, not a TCP problem.

Reading the Output: Timestamps, Numbers and Verbosity

sudo tcpdump -nn -tttt -i eth0 port 443            # human-readable timestamps
sudo tcpdump -e -nn -i eth0                        # include MAC addresses
sudo tcpdump -X -nn -i eth0 -c 3                   # hex + ASCII payload
sudo tcpdump -A -nn -i eth0 -c 3 'tcp port 80'     # ASCII only (HTTP)
sudo tcpdump -vv -nn -i eth0 icmp                  # verbose protocol decode
sudo tcpdump -q -nn -i eth0 -c 100                 # quiet: header lines only
sudo tcpdump -S -nn -i eth0 host 10.0.0.5          # absolute sequence numbers

-nn (combined with the implicit -n) skips both DNS and port-name lookups; without it, a capture on a broken DNS path spends its time on resolver timeouts and prints http and https instead of 80 and 443. -tttt gives wall-clock timestamps, -S replaces relative sequence numbers with absolute ones so you can spot retransmissions by eye, and -e adds the link-layer header — essential when you suspect a MAC learning or VLAN tagging issue rather than an IP one.

Writing Captures to File: Rings, Rotation and Privileges

For anything longer than a few seconds, write to a file and analyse in Wireshark. The ring-buffer options keep a fixed amount of history without filling the disk.

# Rolling capture for Wireshark
sudo tcpdump -nn -i enp3s0 -w /tmp/cap-%Y%m%d-%H%M%S.pcap -G 3600 -W 10 tcp port 80
# Size-based ring: 20 files of 100 MB each
sudo tcpdump -nn -i enp3s0 -w /var/tmp/cap.pcap -C 100 -W 20 -Z tcpdump
# Packet-buffered writes so the file is readable while capturing
sudo tcpdump -nn -i enp3s0 -U -w /tmp/live.pcap
# Read back with the same BPF syntax
sudo tcpdump -nn -r /tmp/live.pcap 'tcp port 443 and host 10.0.0.5'
sudo tcpdump -nn -r /tmp/live.pcap -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

-G 3600 -W 10 means "rotate every 3600 seconds, keep 10 files"; -C 100 -W 20 means "rotate every 100 MB, keep 20 files". Combine them and tcpdump names files with a timestamp you control via strftime in the -w path. Never capture to the root filesystem of a busy host without a size cap: a 10 GbE burst can produce gigabytes in seconds, and a full / causes a much worse incident than the one you were investigating. -Z tcpdump drops root privileges to a designated user once the interface is open.

Practical Scenarios

# DNS queries and responses
sudo tcpdump -i any port 53
# ARP debugging (IP conflicts, resolution failures)
sudo tcpdump -nn -i enp3s0 arp
# Jumbo frames / MTU issues
sudo tcpdump -nn -i enp3s0 'greater 1500'
# Rolling capture for Wireshark
sudo tcpdump -nn -i enp3s0 -w /tmp/cap-%Y%m%d-%H%M%S.pcap -G 3600 -W 10 tcp port 80

A few more that earn their place in a runbook:

# DHCP DORA for a client that will not get an address
sudo tcpdump -nn -i eth0 '(port 67 or port 68) and ether broadcast'
# NTP in and out, with packet counts
sudo tcpdump -nn -c 20 -i any 'udp port 123'
# BGP keepalives and OPEN messages (short frames)
sudo tcpdump -nn -i eth0 'tcp port 179 and len < 90'
# Confirm which VLAN a frame really carries
sudo tcpdump -nn -e -i eth0 'vlan 100 and arp'
# Watch a single session from the first packet
sudo tcpdump -nn -i any -c 20 'host 10.0.0.5 and port 22'
# DHCP relay / gratuitous ARP anomaly hunting
sudo tcpdump -nn -i eth0 'arp and arp[6:2] == 2'

For MTU problems, capture on both ends and compare: if the sender emits fragments or the receiver never sees the large frame, you have a PMTUD black hole, and a greater 1500 filter proves whether the oversized packets actually appear on the wire. For a suspected application problem, the three-way handshake tells you more in three lines than a log file will in three hundred: connection established but application silent points at the process, connection never established points at the network.

Performance, Safety and Permissions on Production Servers

Capture for seconds, not hours. Filter as tightly as the question allows, cap the packet count with -c, and shrink the snaplen to the first 96 or 128 bytes when you only need headers — the payload of a bulk transfer is of no interest and multiplies the CPU and I/O cost. On high-rate links, enlarge the ring buffer and accept that some drops are normal rather than chasing a perfect capture on a live production system.

sudo tcpdump -i eth0 -s 128 -c 2000 -nn -w /tmp/headers.pcap 'tcp port 443'
sudo tcpdump -i eth0 -B 4096 -nn -c 100000 -w /tmp/bulk.pcap

When you must watch a link continuously, do not capture on the server at all — configure a SPAN or TAP port on the switch and capture there, so the production host pays nothing. If you do capture locally, keep the file off the root filesystem, restrict who can run the tool, and remember that captures contain credentials, session cookies and personal data: delete them once the investigation closes.

Troubleshooting tcpdump Itself

Most failures fall into four buckets.

  • Permission denied — the binary lacks cap_net_raw,cap_net_admin and you are not root. Fix with setcap, not by making tcpdump setuid.
  • "No suitable device found" — the named interface does not exist, is down, or the name has a typo; confirm with ip -br link and tcpdump -D.
  • "Syntax error" — the shell consumed your parentheses or pipe characters. Single-quote the whole filter.
  • Packets captured but nothing of interest — the traffic left through a different interface, or NAT rewrote the addresses before capture. Try -i any and filter on the pre-translation addresses.

Finally, watch the summary line that tcpdump prints when it exits. A non-zero "dropped by kernel" count invalidates the capture for timing analysis: the packets you are missing are the ones that would have told you what went wrong.

Related Guides on This Site

For more capture recipes see tcpdump Command Examples: Filters and Practical Packet Capture and Linux Network Troubleshooting: ss, netstat and tcpdump. Move captures into the GUI with Wireshark Capture Filters vs Display Filters and Wireshark TCP: Retransmission vs Out-of-Order vs Loss, and check the link itself with ethtool Linux NIC Diagnostics. For the wider method behind these captures see Linux 网络故障排查方法 and Linux 运维高阶命令实战.

原文链接:https://netstuts.com/tcpdump