fio Disk Benchmarking: Job Files, IO Depth and Latency

fio — the Flexible I/O tester — is the standard tool for measuring storage performance, and it is also one of the easiest tools to misuse. Run it with the wrong queue depth or on a file that fits entirely in page cache and you will produce numbers that look excellent and describe nothing. This guide covers the job file format, the parameters that determine whether a test is meaningful (ioengine, iodepth, direct, runtime) and how to interpret the IOPS, bandwidth and latency figures it prints.

Job file format

fio accepts a huge command line, but anything you intend to reuse belongs in a job file. A job file has a [global] section and one or more named jobs that inherit from it.

; -- start job file --
[global]
ioengine=libaio
direct=1
runtime=60
time_based=1
group_reporting

[randread-4k-qd32]
rw=randread
bs=4k
iodepth=32
size=8g
filename=/dev/nvme0n1

; -- end job file --

fio random-write.fio
fio --output-format=json random-write.fio | jq '.jobs[0].read.iops'

The parameters that decide whether the test means anything

• direct=1 — bypasses the page cache. Without it you are benchmarking RAM. On the other hand, direct=0 is the correct setting when the goal is to model a database that relies on cached reads.
• ioengine — libaio for asynchronous Linux I/O, io_uring on modern kernels, sync or psync for a single-threaded blocking model. Choosing sync and then setting iodepth=32 achieves nothing, because synchronous engines cannot queue.
• iodepth — queue depth. This single number often changes IOPS by an order of magnitude on NVMe. Test at more than one depth (4, 32, 128) rather than reporting one figure.
• bs — block size. 4 KB models database and metadata I/O, 128 KB to 1 MB models streaming and backup workloads. Reporting only 4 KB numbers hides sequential throughput, and only large-block numbers hide IOPS.
• numjobs — parallel workers, which is how you reach high total queue depth from many processes.
• runtime + time_based — run for a fixed wall-clock period instead of until the file is exhausted, so the measurement covers steady state.

Read four files with different queue depths

The cleanest way to see the effect of depth is one job file with per-job overrides.

[global]
ioengine=libaio
direct=1
rw=randread
bs=128k
size=512m
directory=/data1

[file1]
iodepth=4

[file2]
iodepth=32

[file3]
iodepth=8

[file4]
iodepth=16

directory lets fio create and remove its own files, which avoids the "I benchmarked the wrong device" mistake that happens with hand-typed paths.

Reading the output

For each job fio prints separate read and write blocks. The columns that matter:

• IOPS — the headline number for random workloads.
• BW — bandwidth in KiB/s, MiB/s or GiB/s; the meaningful figure for sequential streaming.
• lat (msec/usec) — average, plus percentiles (clat percentiles). Tail latency is what users feel: a device with 0.2 ms average but 40 ms at p99.99 will still stall applications.
• IO depths — the histogram of achieved queue depth. If you asked for 32 and the histogram sits at 1, your engine or block size prevents queuing and the IOPS number is a limitation of the test, not the device.

A few practical cautions

1. Never run destructive write tests against a device holding data. Use size= with a dedicated file, or a scratch LUN.
2. Warm the workload up. Run a short pre-test, then measure, otherwise caching effects dominate the first seconds.
3. Benchmark the same layout you will deploy. A test against a single disk tells you nothing about the RAID or pool you actually run.
4. Watch the CPU. A device that saturates every core while hitting 200k IOPS is CPU-bound, and top will show fio at 100 percent per worker.
5. Correlate with health data before drawing conclusions about a "slow" disk — see SMART disk health monitoring for the counters worth checking, and ZFS zpool and RAIDZ administration for what the storage layer above it does with that performance.

原文链接：https://fio.readthedocs.io/en/latest/fio_doc.html