Linux Network Troubleshooting: ss, netstat and tcpdump Guide

When a Linux server has network problems, the answer is almost always in three tools: ss for socket-level state, tcpdump for packet-level evidence, and netstat on legacy systems. This guide covers the essential commands, filters and interpretation - plus a systematic workflow that separates application, transport and network layer faults, so you stop guessing which layer is broken.

ss: Socket Statistics

ss -tlnp                      # listening TCP sockets with owning process
ss -tunap                     # all TCP/UDP connections with process info
ss -s                         # summary statistics
ss -t state established
ss -t state time-wait
ss -t state close-wait        # often indicates application bugs
ss -tn dport = :443           # filter by destination port
ss -tn dst 192.168.1.50       # filter by address
ss -ti dst 192.168.1.50       # internal TCP info (RTT, cwnd, retrans)

ss reads socket data directly from kernel netlink, making it faster than netstat (which parses /proc/net) and it ships with iproute2 on modern distributions. The flags compose predictably: -t selects TCP, -u UDP, -l listening, -a all, -n numeric (skip DNS and service-name lookups, which is what you want when a resolver is part of the problem), and -p process. Combining them is the point - -n with -p is the difference between waiting thirty seconds for reverse DNS and getting an instant answer.

The filter syntax is where ss beats the older tools. Expressions such as dport = :443, src 10.0.0.0/8, sport > :1024 and state close-wait are evaluated in the kernel, so ss -tn state close-wait returns exactly the stuck sockets without a pipeline of grep and wc. ss -ti is the deepening step: it prints the internal TCP state per socket, including smoothed round-trip time, congestion window, retransmit count and the sender/receiver byte counters. A socket that looks fine on ss -tn but shows a growing retransmit count under ss -ti is a packet-loss problem, not an application problem.

Decoding TCP Socket States

The state column turns a list of sockets into a diagnosis. The states worth recognizing:

• LISTEN - the process is accepting connections. If the service is unreachable locally, the listener is missing or bound to the wrong address.
• ESTAB - an established connection. Many thousands of them is a capacity question, not a fault.
• SYN-SENT - an outgoing connection with no reply yet. Usually a firewall silently dropping packets or a destination that is down.
• TIME-WAIT - the side that closed first waits two MSL (typically 60 s). A large but stable count means a high connection rate; thousands piling up on a single client can exhaust ephemeral ports.
• CLOSE-WAIT - the peer closed and the local application has not. This is nearly always a bug in the application, and a count that grows over time is a leak.
• FIN-WAIT-2 - the local side closed and is waiting for the peer's FIN. Long-lived entries point at a peer that is not completing the handshake teardown.

Counting states is a fast sanity check: ss -tan state time-wait | wc -l, ss -tan state close-wait | wc -l, ss -tan state syn-sent | wc -l. A rising SYN-SENT count with no ESTAB entries is the signature of a one-way path or an egress filter.

Legacy netstat and Equivalents

netstat -tlnp      # TCP listeners with PIDs      (ss -tlnp)
netstat -ulnp      # UDP listeners with PIDs      (ss -ulnp)
netstat -an        # all connections, numeric     (ss -an)
netstat -s         # protocol statistics (TCP retransmits, UDP errors)
netstat -i         # interface statistics
netstat -r         # routing table                (ip route)

netstat is deprecated - it lives in net-tools, which many current distributions no longer install by default - but it still appears in runbooks, and the protocol statistics view has a modern equivalent that is arguably better. ss -s gives the socket summary that netstat -s partially covers, while nstat exposes the kernel's netstat counters as deltas since the last run, which is far more useful for spotting an error rate that is climbing right now. ip -s link replaces netstat -i and ip route replaces netstat -r; both show drop and error counters that netstat -i never did.

tcpdump: Packet Capture

sudo tcpdump -i ens3 -nn -s 0 -w /tmp/capture.pcap port 80
sudo tcpdump -i ens3 -nn host 192.168.1.50
sudo tcpdump -i ens3 -nn 'tcp and dst port 80 and src net 10.0.0.0/8'
sudo tcpdump -i ens3 -nn 'tcp[tcpflags] & (tcp-syn) != 0'   # SYN packets
sudo tcpdump -i ens3 -nn -A -s 0 'tcp port 80' | grep -E 'GET|POST|HTTP/'
tcpdump -nn -r /tmp/capture.pcap

Always apply a BPF filter to limit data volume; use -w file.pcap for long captures and analyze them later in Wireshark. The flags that matter on every capture: -i picks the interface, -nn disables host and port name resolution, -s 0 captures the full frame rather than the historical 68-byte snapshot, -A prints payload as ASCII for text protocols, and -c N stops after N packets so a forgotten capture does not fill the disk. Write to a file whenever the volume is unknown; a pcap that was truncated by a full filesystem is worthless.

On the interface choice, -i any is tempting but changes what you see: the kernel's generic capture path sees traffic once per packet rather than once per ingress and egress, and the link-layer header type differs. Capture on the specific interface when you are investigating a particular direction, including the bridge or VXLAN device rather than its physical parent when the question is about forwarding.

Reading tcpdump Output

A line such as 10.0.0.5.44321 > 10.0.0.9.443: Flags [S], seq 1234, win 64240, options [mss 1460,sackOK,nop,wscale 7], length 0 carries more than it looks. The sequence number and flags tell you where in the handshake the packet sits: [S] is a SYN, [S.] a SYN-ACK, [.] an ACK, [P.] a push with data, [F.] a FIN. Three packets in the right order is a healthy handshake; a SYN with no SYN-ACK within the timeout means the path, the firewall, or the listener is the problem, and the side that never replied is the side to investigate. A SYN-ACK followed by a RST from the client usually means the client has no listener for that port on the return path or the connection was refused by a local policy.

The MSS option is worth reading too. A SYN advertising mss 1460 says the sender expects a 1500-byte MTU; a suspiciously low value such as mss 1400 tells you a tunnel or a policy is already shrinking the path, which is a lead when large transfers fail while small ones succeed. Repeated identical retransmissions of the same sequence number are the clearest packet-loss evidence available, and they are far easier to spot in Wireshark's expert analysis than in a scrolling terminal - capture with tcpdump, analyze with Wireshark.

Advanced Diagnostic Tools

mtr -n 8.8.8.8                          # ping + traceroute combined
nmap -sV 192.168.1.50                   # service version detection
curl -o /dev/null -s -w "HTTP %{http_code} in %{time_total}s\n" https://example.com
openssl s_client -connect example.com:443 -servername example.com
iperf3 -c 192.168.1.50 -t 30 -P 4       # bandwidth test

Each tool answers a question the previous one cannot. mtr shows per-hop loss and latency over time, which distinguishes a lossy intermediate hop from a lossy end host - the former harms nothing if the later hops are clean, the latter is a real fault. nmap -sV proves whether the port is open at all, which resolves the ambiguity of a firewall that drops rather than rejects. The curl -w one-liner separates connection time from transfer time: a long time_connect points at network or TCP handshake problems, a long time_starttransfer points at the application. openssl s_client isolates TLS from everything above it and is the fastest way to prove an expired certificate or an SNI mismatch. iperf3 with four parallel streams measures achievable throughput rather than a single flow's limit, which is what you want when comparing against a link's nominal speed.

Diagnostic Workflow

# 1. Is the service listening?
sudo ss -tlnp | grep :80
# 2. Can you connect locally?
curl -v http://localhost:80
# 3. Is the firewall blocking?
sudo iptables -L -n -v | grep 80
# 4. Can you reach the server at all?
ping -c 3 192.168.1.50
# 5. Is there a path to the server?
traceroute -n 192.168.1.50
# 6. Capture packets to see what is happening
sudo tcpdump -i ens3 -nn host 192.168.1.50 and port 80
# 7. Check connection states for anomalies
ss -tn state close-wait | wc -l     # many CLOSE-WAIT = app bug
ss -tn state time-wait | wc -l      # many TIME-WAIT = high connection rate

The order is the whole method. Each step halves the search space: step 1 proves the socket exists; step 2 separates the application from the network by testing over loopback; step 3 rules in or out a local filter; step 4 proves layer 3 reachability; step 5 proves the path; step 6 gives you the actual packets when steps 2-5 disagree with each other; step 7 catches the systemic cases where no single packet is wrong but the aggregate is. Skipping straight to tcpdump is the most common mistake - a capture of the wrong interface, taken at the wrong time, tells you nothing while looking authoritative.

Common Scenarios and What They Look Like

• SYN sent, no reply, no RST. Something on the path is silently dropping. Compare a capture on the client's egress interface with one on the server's ingress interface: if the SYN appears on the client only, the path or an intermediate filter is the cause.
• Connection refused instantly. A RST came back, so the path is fine and there is no listener, or the listener is bound to a different address (127.0.0.1 instead of 0.0.0.0). ss -tlnp on the server settles it in one line.
• Handshake completes, then hangs. Classic MTU or MSS black hole. Small packets work, large ones vanish. Test with ping -M do -s 1472 and capture to see whether the DF-bit packet ever gets a reply.
• Slow but not broken. Look at ss -ti for retransmits and at ip -s link for interface drops; a duplex mismatch or a saturated uplink shows up as drops and errors on the interface, not in the application log.
• Intermittent failures under load. Check for a full accept queue (ss -lnt shows Send-Q as the backlog for LISTEN sockets) and for TIME-WAIT accumulation on the client side.

Start with ss for a socket overview, drill into tcpdump for packet evidence, and correlate socket state counts with kernel metrics when the problem is systemic.

Related articles

Going deeper on these tools: tcpdump packet capture guide: BPF filters and troubleshooting, tcpdump command examples and practical captures, mtr command guide: diagnosing packet loss and latency, and ethtool explained: Linux NIC diagnostics.

原文链接：https://kindatechnical.com/linux-administration/network-diagnostics-ss-netstat-tcpdump.html