Linux Performance Troubleshooting: Tools and Commands

When a Linux server slows down, guessing wastes precious time - a systematic method finds the bottleneck in minutes. This practical guide applies the USE method (Utilization, Saturation, Errors) to Linux performance troubleshooting, starting with a 60-second checklist of one-line diagnostics, then drilling into CPU, memory, disk I/O and network with the right tool for each layer: top, mpstat, vmstat, free, iostat, sar, ss and dmesg. Every command is ready to copy and run on your server.

The USE Method: A Structured Framework

For every resource (CPU, memory, disk, network), check: Utilization (how busy is it), Saturation (how much extra work is queued), and Errors (any error conditions). For CPU: utilization with top/mpstat, saturation with the vmstat run queue, errors with dmesg/perf. For disk: utilization with iostat, saturation with iostat aqu-sz, errors with dmesg/smartctl.

Interpreting Load Average Correctly

The load average is the most misread number in Linux. It is not a percentage of CPU, and it is not a measure of CPU at all. On Linux it counts the number of processes that are either runnable or blocked in uninterruptible sleep (state D), averaged over one, five and fifteen minutes. That means a server with four cores and a load average of 8 is not necessarily out of CPU — it may be waiting on a disk that has nothing to do with CPU at all.

nproc                    # how many runnable slots you actually have
uptime                   # load averages
cat /proc/loadavg        # load plus runnable/total and last PID

Treat the load average as a ratio to nproc. Below 1.0× you have headroom; sustained above roughly 1.0× per core means something is queuing. Read the trend, not the instant: if the one-minute figure is rising above the five-minute figure (8.52 > 4.31 > 2.15), the situation is deteriorating right now; if it is falling, you are looking at the tail of an event. Fifteen-minute load that is high while the one-minute figure is low is history, not an incident.

CPU versus I/O wait is what makes the number actionable. A process blocked on disk drives the load average up while the CPU sits idle — the classic "high load, low CPU" pattern. Confirm with vmstat: if r is small but b is large, the bottleneck is I/O; if r consistently exceeds nproc, you are genuinely CPU-bound.

In virtual machines there is one more caveat: st (steal time) is CPU the hypervisor took from you. A guest with high load, low utilisation and non-zero steal is not misconfigured — it is short of allocated CPU, and the fix is the instance size, not the guest.

Building a Baseline Before You Need One

Troubleshooting from zero is guesswork because you have no idea what normal looks like. Collect a baseline while the server is healthy: sysstat writes one sample every ten minutes into /var/log/sysstat, giving you a week of history to compare against.

sudo apt install sysstat            # dnf install sysstat on RHEL
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat

# today vs. a quiet day, per-CPU
sar -u -f /var/log/sysstat/sa$(date +%d)
sar -r -f /var/log/sysstat/sa$(date +%d)      # memory
sar -n DEV -f /var/log/sysstat/sa$(date +%d)  # network

Step 1: The 60-Second Checklist

uptime                              # 1. Load averages
dmesg -T | tail                     # 2. Kernel errors
vmstat 1 5                          # 3. CPU, memory, I/O overview
mpstat -P ALL 1 3                   # 4. Per-CPU breakdown
iostat -xz 1 3                      # 5. Disk I/O
free -h                             # 6. Memory
sar -n DEV 1 3                      # 7. Network throughput
ss -s                               # 8. Connection summary
top -bn1 | head -20                 # 9. Top processes
df -h                               # 10. Disk space

Compare the load averages: if they are rising (8.52 > 4.31 > 2.15) the system is getting busier. Load average should stay below the number of CPU cores (nproc).

CPU Deep Dive: Beyond top

top tells you a process is hot; it rarely tells you why. The first split that matters is user versus system time. High us with low sy is application work — algorithm cost, thread contention, garbage collection. High sy is kernel work — syscall volume, networking, context switching, or a container doing a great deal of read()/write() for small payloads. Getting this split right removes half the candidates immediately.

mpstat -P ALL 2 5        # per-CPU: spot a single saturated core
pidstat -u 1 5           # per-process CPU, including %usr vs %system
pidstat -w 1 5           # context switches per process - a runaway wakeup pattern
vmstat 2 10              # cs column: system-wide context switch rate
ps -eo pid,ppid,cmd,%cpu,etime --sort=-%cpu | head -15

A single 100% core on a 16-core box is invisible in top's aggregate view and is the usual cause of a latency complaint on a multi-threaded service. Always check the per-CPU breakdown before concluding the CPU is fine. Similarly, a high cs (context switch) rate with modest us points at lock contention or thread thrashing rather than raw compute, and pidstat -w narrows it to a specific process.

%iowait deserves a warning. It is not a measure of how busy a disk is; it is the percentage of time a CPU was idle and at least one I/O was outstanding. On a system with sixteen cores and one slow disk, the true iowait is almost always understated, and on a busy system it is systematically misleading. Use iostat for the disk itself and treat %iowait as a hint that something is waiting, never as proof of disk saturation.

sudo perf top -g                       # live kernel + user hotspots
sudo perf record -F 99 -g -p $(pgrep -n mysqld) -- sleep 30
sudo perf report --stdio | head -40
sudo strace -c -f -p $(pgrep -n nginx) # syscall cost summary

Step 2: CPU Troubleshooting

top -o %CPU                 # top processes by CPU; press 1 for per-core, c for cmdline
htop                        # interactive tree view
mpstat -P ALL 2 5           # per-CPU utilization; watch %iowait and %steal
ps aux --sort=-%cpu | head -20
vmstat 2 10                 # if 'r' column exceeds CPU count, CPU is saturated

In vmstat output: r = run queue, us = user CPU, sy = system CPU, id = idle, wa = I/O wait, st = stolen time (VM contention).

Memory Deep Dive: Page Cache, Swap and the OOM Killer

On Linux, free memory is wasted memory — the kernel deliberately fills it with page cache, and that cache is reclaimed instantly when a process needs RAM. So "free memory is low" is normal and healthy. What actually matters is three different numbers that look similar and mean completely different things:

free -h
cat /proc/meminfo | grep -E 'MemAvailable|MemFree|Cached|Dirty|SwapCached|Committed_AS'
vmstat 2 5      # watch the si/so columns
cat /proc/$(pgrep -n mysqld)/status | grep -E 'VmRSS|VmSwap'

MemFree is memory doing nothing. Cached is page cache, reclaimable on demand. MemAvailable is the kernel's estimate of what a new workload could allocate without swapping — this is the number to watch, and it is the one free -h shows as available. A server with 1 GB free and 40 GB cached is not short of memory; it is being used well. A server with 1 GB available and active swap traffic is in trouble.

Swapping is the real signal. Non-zero si/so in vmstat means the kernel is paging to disk, which translates directly into latency: sustained swapping is a memory saturation event even if nothing has died yet. Per-process swap usage names the workload being evicted:

for p in /proc/[0-9]*; do
  awk '/^Name|^VmSwap/{printf "%s ", $2}' "$p/status" 2>/dev/null; echo
done | awk '$NF+0 > 0' | sort -k2 -hr | head -10

When memory is genuinely exhausted, the OOM killer selects a victim using oom_score, which is driven largely by the process's memory footprint and adjustable through oom_score_adj. It is not a random choice and it is not always the culprit that dies — a leaking cache may take out a critical daemon instead. Check for it explicitly:

sudo dmesg -T | grep -i -E 'oom|killed process'
journalctl -k --since '24 hours ago' | grep -i oom

Protect critical processes with a negative adjustment (-1000 makes a process ineligible) and cap the real consumers so one runaway cannot take the host down:

sudo systemctl edit myservice     # add: [Service] OOMScoreAdjust=-500
sudo systemctl set-property myservice.service MemoryMax=2G

Step 3: Memory Troubleshooting

Low free memory is normal on Linux - the kernel uses free memory for caching. Look instead at free -h swap usage and vmstat swapping: sustained swap-in/swap-out means memory saturation. Check dmesg for OOM (Out Of Memory) killer events that kill processes when memory is exhausted.

Disk I/O Deep Dive: Reading iostat Without Guessing

iostat produces a wall of numbers, and the ones people reach for are frequently the wrong ones. The essential fields:

iostat -xz 2 5
# r/s, w/s    - read and write operations per second (IOPS)
# rkB/s, wkB/s- throughput in kilobytes per second
# await       - average latency per I/O, milliseconds (queue + service)
# r_await, w_await - split read and write latency
# aqu-sz      - average queue length (older versions: avgqu-sz)
# %util       - percent of time the device had at least one I/O in flight
# rrqm/s, wrqm/s - requests merged, a sign of sequential access

%util is the most abused metric. On a single spinning disk, 100% is genuine saturation. On an SSD or a RAID volume built from many devices, 100% %util may still be well within the device's capacity, because the device can service many parallel requests. Judge saturation by await and aqu-sz against your own baseline latency: if await has tripled and queue depth is climbing, you are queuing, whatever %util says.

Read the numbers together. High w/s with tiny wkB/s means many small writes — a committing database or fsync-heavy logging. High wkB/s with low w/s means large sequential writes, usually a benign backup. High await with low throughput is the worst combination and points at a failing device or an overloaded backend.

sudo ioping -c 10 /var/lib/mysql        # latency in the actual working directory
sudo smartctl -a /dev/sda | grep -E 'Reallocated|Pending|Uncorrectable|Wear'
sudo hdparm -tT /dev/sda                # cached and buffered read speed
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL,ROTA

Attribute the load to a workload before changing anything. iotop -oPa ranks processes by current I/O, and /proc/<pid>/io gives cumulative bytes per process. Then check dmesg, which carries the warnings that matter most: I/O errors, ata link resets, a filesystem remounted read-only, and task blocked for more than 120 seconds. Those messages are strong evidence a drive is dying. Never replace a device on %util alone — confirm against SMART attributes first, per smartctl and smartd disk health monitoring.

Step 4: Disk I/O Troubleshooting

iostat -x 1              # extended device stats: %util, await, aqu-sz
iostat -dxctm 1          # throughput, IOPS, queue size and latency

High %util or large aqu-sz (average queue size) indicates disk saturation. Correlate with vmstat wa column and dmesg for I/O errors; use smartctl to check drive health on suspect devices.

Network Deep Dive: Latency, Drops and Sockets

Network issues masquerade as server slowness more often than people expect: a server whose application threads block on a slow peer looks like a CPU or memory problem from the inside. Start with the counters and the socket table rather than packet captures.

ss -s                                    # socket summary by state
ss -tanp | awk '{print $1}' | sort | uniq -c | sort -rn
ss -tanp state time-wait | wc -l         # TIME-WAIT accumulation
ss -lntp                                 # listening sockets and their backlogs
nstat -az                                # all kernel network counters, cumulative
sar -n DEV 2 3                           # per-interface throughput
ip -s link show eth0                     # errors, drops, overruns per interface

The state distribution is the highest-value single view. A large TIME-WAIT count is normal on a busy web server but signals short-lived connections you could pool. Large SYN-RECV or ESTABLISHED counts alongside latency suggest a SYN flood, a full listen backlog (Send-Q in ss -lnt) or downstream slowness. And a large CLOSE-WAIT count is always an application bug: the peer closed and your process never called close(), so descriptors leak until it runs out.

Interface counters separate local faults from upstream ones. Climbing rx_errors, rx_missed_errors or rx_dropped on ip -s link or ethtool -S eth0 means packets are lost before the application sees them — a full ring buffer, a duplex mismatch, a CPU unable to service interrupts, or an offload setting the switch disagrees with. Check the buffer and offload state:

ethtool -g eth0                          # ring buffer sizes
ethtool -S eth0 | grep -E 'drop|error|miss|discard'
sudo ethtool -K eth0 gro on tso on gso on
cat /proc/net/dev                        # per-interface bytes, packets, errors, drops
sysctl net.core.rmem_max net.ipv4.tcp_rmem

Latency needs a different tool from throughput: mtr shows where loss and delay begin along the path, while ping -f and iperf3 separate delay from capacity. When the loss is specific to one flow, move to packet capture — tcpdump command examples and capture filters, Wireshark retransmission versus out-of-order analysis and ss, netstat and tcpdump cover the details.

Step 5: Network Troubleshooting

sar -n DEV 1 3          # per-interface throughput
ss -s                   # connection summary (replaces netstat)
ip -s link              # per-interface error and drop counters

High error/drop counters on ip -s link point to duplex mismatches, cable issues or ring-buffer exhaustion. Combining these layers lets you isolate the bottleneck: top finds the hot process, vmstat confirms swapping, and iostat proves whether disk is the culprit.

Pressure Stall Information and Modern Tracing

Kernel 4.20 and later expose Pressure Stall Information, which answers the question the load average cannot: how much wall-clock time did work actually lose to contention? PSI reports the percentage of time tasks were stalled waiting for CPU, memory or I/O, and it is the single most useful modern signal for "the server feels slow".

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

# watch it live: some avg10, total, and the per-cgroup breakdown
watch -n1 'cat /proc/pressure/io'

Read some as "at least one task stalled" and full as "every runnable task stalled" — the latter is a genuine outage of that resource; a memory full above zero means allocations are being throttled by reclaim. PSI also works per cgroup, so in a container cat /sys/fs/cgroup/<path>/io.pressure shows whether your container or its neighbours own the device. cgroup v2's memory.events carries the oom_kill counter — the definitive record of how many times the limit killed a process.

Beyond counters, the modern tool is eBPF — low-impact kernel probes that can be run in production with care:

sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'
sudo execsnoop-bpfcc                    # short-lived processes, from bcc
sudo biosnoop-bpfcc                     # per-I/O latency with process attribution

Run these for seconds, not minutes. bpftrace one-liners for production tracing has a wider library, including tail-latency histograms.

Five-Minute Triage for Common Scenarios

1. Everything is slow. Run uptime, vmstat 1 5, iostat -xz 1 3, free -h. High r with low wa means CPU; high b and wa means disk; non-zero si/so means memory.
2. One service is slow, the host looks fine. pidstat -u -w -d -r -p PID 1 5. Per-process CPU, context switches, I/O and page faults in one view, followed by perf top or strace -c if the numbers do not explain it.
3. Latency spikes, averages look normal. Averages hide tails — use bpftrace histograms, PSI, or application percentiles. Check for periodic events (cron, log rotation, backups, updatedb) with journalctl --since.
4. Load average high, CPU idle. Almost always I/O or uninterruptible sleep. vmstat's b column, ps -eo stat,pid,cmd | grep ' D', and dmesg for storage warnings.
5. Connections timing out under load. ss -s, check for CLOSE-WAIT or SYN-RECV accumulation, verify the listen backlog, then nstat -az for listen-overflow counters and sysctl net.core.somaxconn.
6. Memory creeping up for days. Track VmRSS and slab, then profile the suspect process. Confirm with memory.events whether the cgroup is being throttled before the OOM killer fires.

Two rules shorten every investigation: change one thing at a time, and prefer evidence that names a culprit over a number that merely says something is wrong.

Related Reading

• Linux 网络故障排查：从网卡到路由的定位思路
• Linux 运维高阶命令实战：五大场景故障排查指南
• 适合初学者的 17 个最佳 Linux 网络和故障排除命令
• bpftrace 生产环境一行命令追踪
• smartctl 与 smartd 磁盘健康监控
• tcpdump 命令示例与抓包过滤器
• mtr 诊断丢包与延迟

Install the Toolset

sudo apt install sysstat procps htop iotop iftop iperf3 strace lsof smartmontools mtr-tiny bpfcc-tools bpftrace
sudo dnf install sysstat procps-ng htop iotop iftop iperf3 perf strace lsof smartmontools mtr bcc-tools bpftrace

Install them before you need them — fetching packages on a degraded machine with a full disk and no DNS turns a five-minute diagnosis into a twenty-minute one.

原文链接：https://oneuptime.com/blog/post/2026-02-20-linux-performance-troubleshooting/view