bpftrace One-Liners for Production Linux Debugging

Once a problem is “sudden but not reproducible”, static tools stop helping. bpftrace gives you one-line answers from a live kernel without a reload, a restart or a debug build: which process is opening that file, which syscall is slow, which PID is issuing all the disk I/O. This article collects the one-liners that earn their place on a production host and the prerequisites that decide whether they run at all.

Prerequisites

• Kernel 4.9 or newer (5.x+ for the widest probe coverage) and a distro package such as bpftrace; on Debian/Ubuntu apt install bpftrace, on RHEL family dnf install bpftrace.
• BTF information (/sys/kernel/btf/vmlinux) — without it, kprobe arguments cannot be introspected and some scripts need manual header includes.
• Root or CAP_BPF/CAP_PERFMON; on locked-down hosts, kernel.unprivileged_bpf_disabled and lockdown mode will block you.
• Probe budget: use -l to check a probe exists before building a script, and be careful with high-frequency probes such as tracepoint:raw_syscalls:sys_enter on a busy server — aggregate with counts rather than printing every event.

The one-liners

# Which process opens which file (find the config file a service reads)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'

# Syscall volume per process - spot the noisy one
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Bytes read per process
bpftrace -e 'tracepoint:syscalls:sys_exit_read /args.ret > 0/ { @[comm] = sum(args.ret); }'

# Read-size distribution (histogram = what the application actually asks for)
bpftrace -e 'tracepoint:syscalls:sys_exit_read { @[comm] = hist(args.ret); }'

# Per-second syscall rate, printed live
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @ = count(); } interval:s:1 { print(@); clear(@); }'

# Who is issuing block I/O and how big it is
bpftrace -e 'tracepoint:block:block_rq_issue { printf("%d %s %d bytes\n", pid, comm, args.bytes); }'

# Page faults per process (memory pressure triage)
bpftrace -e 'software:faults:1 { @[comm] = count(); }'

# User-level stack profile of one PID (replace perf top with a histogram)
bpftrace -e 'profile:hz:99 /pid == 189/ { @[ustack] = count(); }'

Read the histograms, not the totals. A read-size histogram that is bimodal — lots of 4 KB reads and a separate hump at 1 MB — is the signature of a process that mixes buffered and direct I/O, and it explains wildly inconsistent latency far better than an average.

Tracing latency, not just counts

# Time the openat() calls and bucket the latency
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_openat /@start[tid]/ {
  @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]);
}'

# Trace a specific kernel function and its return value
bpftrace trace.bt   # kprobe/kretprobe pair, e.g. vfs_open + return

# Syscalls slower than 10 ms, with the command name
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @s[tid] = nsecs; }
tracepoint:syscalls:sys_exit_* /@s[tid] && (nsecs - @s[tid]) > 10000000/ {
  printf("%s %s %dms\n", comm, probe, (nsecs - @s[tid])/1000000); delete(@s[tid]); }'

Keep one-off tracers in a .bt file rather than retyping them — the ones worth keeping are the ones you will need again at 03:00 during an incident. bpftrace is also a plain language: probes, filters in / /, actions in { }, and maps (@name) that are printed on exit, which is what makes these scripts readable under pressure.

Related reading: Linux TCP buffer and backlog tuning, ethtool diagnostics and using mtr to diagnose packet loss.

原文链接：https://bpftrace.org/one-liners