Linux Performance Troubleshooting: Tools and Commands

2026年8月25日 LOG admin

When a Linux server slows down, guessing wastes precious time - a systematic method finds the bottleneck in minutes. This practical guide applies the USE method (Utilization, Saturation, Errors) to Linux performance troubleshooting, starting with a 60-second checklist of one-line diagnostics, then drilling into CPU, memory, disk I/O and network with the right tool for each layer: top, mpstat, vmstat, free, iostat, sar, ss and dmesg. Every command is ready to copy and run on your server.

The USE Method: A Structured Framework

For every resource (CPU, memory, disk, network), check: Utilization (how busy is it), Saturation (how much extra work is queued), and Errors (any error conditions). For CPU: utilization with top/mpstat, saturation with the vmstat run queue, errors with dmesg/perf. For disk: utilization with iostat, saturation with iostat aqu-sz, errors with dmesg/smartctl.

Step 1: The 60-Second Checklist

uptime                              # 1. Load averages
dmesg -T | tail                     # 2. Kernel errors
vmstat 1 5                          # 3. CPU, memory, I/O overview
mpstat -P ALL 1 3                   # 4. Per-CPU breakdown
iostat -xz 1 3                      # 5. Disk I/O
free -h                             # 6. Memory
sar -n DEV 1 3                      # 7. Network throughput
ss -s                               # 8. Connection summary
top -bn1 | head -20                 # 9. Top processes
df -h                               # 10. Disk space

Compare the load averages: if they are rising (8.52 > 4.31 > 2.15) the system is getting busier. Load average should stay below the number of CPU cores (nproc).

Step 2: CPU Troubleshooting

top -o %CPU                 # top processes by CPU; press 1 for per-core, c for cmdline
htop                        # interactive tree view
mpstat -P ALL 2 5           # per-CPU utilization; watch %iowait and %steal
ps aux --sort=-%cpu | head -20
vmstat 2 10                 # if 'r' column exceeds CPU count, CPU is saturated

In vmstat output: r = run queue, us = user CPU, sy = system CPU, id = idle, wa = I/O wait, st = stolen time (VM contention).

Step 3: Memory Troubleshooting

Low free memory is normal on Linux - the kernel uses free memory for caching. Look instead at free -h swap usage and vmstat swapping: sustained swap-in/swap-out means memory saturation. Check dmesg for OOM (Out Of Memory) killer events that kill processes when memory is exhausted.

Step 4: Disk I/O Troubleshooting

iostat -x 1              # extended device stats: %util, await, aqu-sz
iostat -dxctm 1          # throughput, IOPS, queue size and latency

High %util or large aqu-sz (average queue size) indicates disk saturation. Correlate with vmstat wa column and dmesg for I/O errors; use smartctl to check drive health on suspect devices.

Step 5: Network Troubleshooting

sar -n DEV 1 3          # per-interface throughput
ss -s                   # connection summary (replaces netstat)
ip -s link              # per-interface error and drop counters

High error/drop counters on ip -s link point to duplex mismatches, cable issues or ring-buffer exhaustion. Combining these layers lets you isolate the bottleneck: top finds the hot process, vmstat confirms swapping, and iostat proves whether disk is the culprit.