bpftrace One-Liners for Production Linux Debugging

Once a problem is “sudden but not reproducible”, static tools stop helping. bpftrace gives you one-line answers from a live kernel without a reload, a restart or a debug build: which process is opening that file, which syscall is slow, which PID is issuing all the disk I/O. This article collects the one-liners that earn their place on a production host, the prerequisites that decide whether they run at all, and the habits that keep them safe to run on a system serving traffic.

Prerequisites

• Kernel 4.9 or newer (5.x+ for the widest probe coverage) and a distro package such as bpftrace; on Debian/Ubuntu apt install bpftrace, on RHEL family dnf install bpftrace.
• BTF information (/sys/kernel/btf/vmlinux) — without it, kprobe arguments cannot be introspected and some scripts need manual header includes.
• Root or CAP_BPF/CAP_PERFMON; on locked-down hosts, kernel.unprivileged_bpf_disabled and lockdown mode will block you.
• Probe budget: use -l to check a probe exists before building a script, and be careful with high-frequency probes such as tracepoint:raw_syscalls:sys_enter on a busy server — aggregate with counts rather than printing every event.

Check the Prerequisites Before You Debug the Debugger

Almost every “bpftrace does not work here” is one of four things, and all four are visible in a couple of commands:

# 1. Is the kernel new enough and is BPF compiled in?
uname -r
zgrep -E 'CONFIG_BPF=|CONFIG_BPF_SYSCALL=|CONFIG_DEBUG_INFO_BTF=' /proc/config.gz \
  || grep -E 'CONFIG_BPF=|CONFIG_BPF_SYSCALL=|CONFIG_DEBUG_INFO_BTF=' /boot/config-$(uname -r)

# 2. Is BTF present? Without this file, kprobe argument access is guesswork.
ls -l /sys/kernel/btf/vmlinux

# 3. Is unprivileged BPF disabled (2 = disabled at boot, good default on servers)?
cat /proc/sys/kernel/unprivileged_bpf_disabled

# 4. Which version, and which probe types does it support?
bpftrace --version
bpftrace -l 'tracepoint:syscalls:sys_enter_*' | head

If BTF is missing because the kernel was built without CONFIG_DEBUG_INFO_BTF, kprobes that reference arguments such as args->filename will fail to compile; tracepoints are unaffected because their argument layout is stable ABI. That is the single best reason to prefer tracepoints when you are writing something you want to run on a fleet of mixed distributions years from now.

Two runtime blockers are worth knowing by name. kernel.unprivileged_bpf_disabled=2 means only processes with the right capability can load BPF at all — that is a security control, not a bug, and the fix is to run with sudo or grant CAP_BPF, never to disable the sysctl on a production host. Kernel lockdown in confidentiality mode will refuse BPF probes outright; mokutil --sb-state and dmesg | grep -i lockdown tell you whether you are fighting secure boot policy rather than syntax.

The one-liners

# Which process opens which file (find the config file a service reads)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'

# Syscall volume per process - spot the noisy one
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Bytes read per process
bpftrace -e 'tracepoint:syscalls:sys_exit_read /args.ret > 0/ { @[comm] = sum(args.ret); }'

# Read-size distribution (histogram = what the application actually asks for)
bpftrace -e 'tracepoint:syscalls:sys_exit_read { @[comm] = hist(args.ret); }'

# Per-second syscall rate, printed live
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @ = count(); } interval:s:1 { print(@); clear(@); }'

# Who is issuing block I/O and how big it is
bpftrace -e 'tracepoint:block:block_rq_issue { printf("%d %s %d bytes\n", pid, comm, args.bytes); }'

# Page faults per process (memory pressure triage)
bpftrace -e 'software:faults:1 { @[comm] = count(); }'

# User-level stack profile of one PID (replace perf top with a histogram)
bpftrace -e 'profile:hz:99 /pid == 189/ { @[ustack] = count(); }'

Read the histograms, not the totals. A read-size histogram that is bimodal — lots of 4 KB reads and a separate hump at 1 MB — is the signature of a process that mixes buffered and direct I/O, and it explains wildly inconsistent latency far better than an average.

Process and File Tracing

These four answer the questions that come up most often during an incident: what just started, what is opening files, what failed, and which filesystem is busy.

# Every process exec with its arguments - the "what started at 03:07" tool
bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
  printf("%d %s: ", pid, comm); join(args.argv); printf("\n"); }'

# Open failures only, with the errno - catches permission and missing-path bugs
bpftrace -e 'tracepoint:syscalls:sys_exit_openat /args.ret < 0/ {
  printf("%s failed %d (%s)\n", comm, args.ret, str(args.filename)); }'

# Which PID is hammering one specific path
bpftrace -e 'tracepoint:syscalls:sys_enter_openat /str(args.filename) == "/etc/resolv.conf"/ {
  printf("%d %s\n", pid, comm); }'

# Filesystem operations per mount point, as a count map
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[str(args.filename)] = count(); }'
bpftrace -e 'vfs_statfs { @[comm] = count(); }'

The failed-open one-liner is the one to keep in muscle memory. A service that reports “config not found” in its own logs usually tried four paths, three of which failed for different reasons, and the errno tells you whether it was a permission problem (-13) or a genuinely missing file (-2).

Disk and Filesystem Latency

iostat tells you the device is slow; bpftrace tells you which process made it slow and with what queue depth. A block latency histogram is the difference between “the array is busy” and “one backup job is issuing 1 MB synchronous writes”.

# Read/write latency distribution in microseconds
bpftrace -e 'tracepoint:block:block_rq_issue { @start[args.cookie] = nsecs; }
tracepoint:block:block_rq_complete /@start[args.cookie]/ {
  @us = hist((nsecs - @start[args.cookie]) / 1000); delete(@start[args.cookie]); }'

# Bytes and operation type per process, so you can see read vs write mix
bpftrace -e 'tracepoint:block:block_rq_issue {
  @[comm, args.rwbs] = hist(args.bytes); }'

# VFS-level call counts per process - the cheap way to spot a syscall storm
bpftrace -e 'kprobe:vfs_read, kprobe:vfs_write { @[comm, probe] = count(); }'

Tracing latency, not just counts

# Time the openat() calls and bucket the latency
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_openat /@start[tid]/ {
  @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]);
}'

# Trace a specific kernel function and its return value
bpftrace trace.bt   # kprobe/kretprobe pair, e.g. vfs_open + return

# Syscalls slower than 10 ms, with the command name
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @s[tid] = nsecs; }
tracepoint:syscalls:sys_exit_* /@s[tid] && (nsecs - @s[tid]) > 10000000/ {
  printf("%s %s %dms\n", comm, probe, (nsecs - @s[tid])/1000000); delete(@s[tid]); }'

Keep one-off tracers in a .bt file rather than retyping them — the ones worth keeping are the ones you will need again at 03:00 during an incident. bpftrace is also a plain language: probes, filters in / /, actions in { }, and maps (@name) that are printed on exit, which is what makes these scripts readable under pressure.

The kernel-side counterpart to a latency histogram is finding out where time goes when no syscall is slow — if the process is runnable but not running, the problem is CPU scheduling, not I/O:

# Run-queue latency: time from wakeup to actually getting a CPU
bpftrace -e 'tracepoint:sched:sched_wakeup { @wake[args.pid] = nsecs; }
tracepoint:sched:sched_switch /@wake[args.next_pid]/ {
  @runq[args.next_comm] = hist((nsecs - @wake[args.next_pid]) / 1000);
  delete(@wake[args.next_pid]); }'

# Off-CPU time: how long threads spend blocked, grouped by command
bpftrace -e 'tracepoint:sched:sched_switch { @out[args.next_pid] = nsecs; }
tracepoint:sched:sched_switch /@out[args.prev_pid]/ {
  @offcpu[args.prev_comm] = sum(nsecs - @out[args.prev_pid]) / 1000000;
  delete(@out[args.prev_pid]); }'

Network and Application Probes

# TCP retransmissions, the useful “the network is lying” counter
bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[comm] = count(); }'

# Every connect() attempt with the target address, in readable form
bpftrace -e 'kprobe:tcp_connect /comm == "nginx"/ { printf("%d %s connecting\n", pid, comm); }'

# Syscall counts and latency spread for one PID only
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 4123/ { @[args.id] = count(); }'
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 4123/ { @s[tid] = nsecs; }
tracepoint:raw_syscalls:sys_exit /@s[tid]/ { @us = hist((nsecs - @s[tid])/1000); delete(@s[tid]); }'

# Static user-space probes in an application that ships USDT/SystemTap markers
bpftrace -l 'usdt:/usr/sbin/postgres:*'
bpftrace -e 'usdt:/usr/sbin/postgres:* { printf("%s %s\n", probe, comm); }'

Filtering by pid or comm is not just cosmetic. A tracer that fires on every event on a box doing 50 000 syscalls per second costs real CPU; the same tracer restricted to the suspect process costs nothing measurable. Always narrow the probe before widening the window.

How bpftrace Compares with the Other Tracers

• strace stops the traced process twice per syscall, which can slow a workload by orders of magnitude. bpftrace attaches a probe and keeps the kernel on its normal fast path — the overhead is real but it is a percentage, not a multiple.
• perf is better for sampling flame graphs and hardware counters over long periods; bpftrace is better for asking a specific question about a specific function or argument right now.
• ftrace through /sys/kernel/debug/tracing is the fallback that is always there, with no extra packages, but its filter language is awkward and the output is raw text.
• SystemTap has richer language features and long-standing kernel-version support; bpftrace is the one that is already packaged in every current distribution.

Running Them Safely on a Production Host

• Start with counts, then narrow. A count() map over ten seconds tells you where to look; a printf() on every event on a busy box can fill a disk in minutes.
• Raise the buffer and reduce wakeups. -b sets the per-CPU perf ring buffer size and -B selects the buffering mode; on high-frequency probes the default can drop events, and a tracer that silently drops events produces confident wrong answers.
• Use a time limit. Wrap long captures with timeout 30 bpftrace ... or use the -c flag to run a command and exit when it finishes, so a forgotten script cannot run until the next reboot.
• Watch for lost events. bpftrace reports dropped events at the end of a run; if the number is not zero, the map you are reading is incomplete.
• Never leave a debug tracer running. bpftrace exits cleanly on Ctrl-C and prints its maps on the way out, which is the intended workflow: narrow window, capture, exit, read the histogram.
• Record before you change. Save the raw output next to the ticket. A histogram taken during the incident is evidence; one taken afterwards is an anecdote.
• Check for existing tooling first. The BCC and bpftrace tool packages ship pre-written versions of most of these one-liners (execsnoop, opensnoop, biolatency, tcpretrans, runqlat) which are better tested than anything you will type at 03:00.

A Practical Triage Order

When a Linux host is slow and nothing in the logs explains it, work in this order and stop when the evidence points at a component:

1. CPU scheduling — a run-queue latency histogram separates “not enough CPU” from “blocked on I/O”.
2. Disk — block latency plus the per-process byte count separates a slow array from one greedy writer.
3. Filesystem — read-size histograms and open-failure counts find the application doing pathological I/O.
4. Network — retransmit counts and per-process connect attempts for anything network-facing; the sysctl side of that is covered in Linux TCP buffer and backlog tuning.
5. Application — once the kernel looks clean, the answer is inside the process, and USDT probes or a stack profile are the way in.

What to Suspect When It Does Not Run

• No probes found — the probe name is wrong for this kernel. Confirm with bpftrace -l '*vfs_open*'; function names change between kernel versions, tracepoint names much less so.
• Error: unknown type on a kprobe argument — BTF is missing, as described above. Switch to the tracepoint equivalent.
• Operation not permitted — you are not root and lack CAP_BPF, or unprivileged BPF is disabled. Escalate properly instead of relocating the problem.
• Kernel lockdown is in effect — secure boot policy. There is no workaround worth using on a production host.
• Everything runs but the map is empty — the filter is wrong, the process name is truncated to 15 characters in comm, or the event genuinely is not happening and your theory was wrong. That is still a result.

Related reading: Linux TCP buffer and backlog tuning, ethtool diagnostics, using mtr to diagnose packet loss, ss, netstat and tcpdump, and for the container side of the same problem set, Calico versus Cilium versus Flannel and LVM thin provisioning.

原文链接：https://bpftrace.org/one-liners