io_uring vs libaio: Storage Performance on NVMe

Storage performance on modern NVMe devices is no longer limited by the media but by the software path around it: each I/O through libaio costs a system call, a context switch, and a copy of the submission structure. io_uring was designed to remove exactly that overhead — submission and completion queues live in shared memory rings, so a thread can submit many I/Os with no syscall at all in the common case. The performance literature on the subject is consistent on where the gains appear and where they do not, and knowing that boundary is what stops a migration from being wasted effort. This guide covers the mechanism and the measured behaviour.

Why system calls dominate

At queue depth 1 with a fast SSD, the kernel entry/exit for each I/O can account for a large share of the total latency. As devices reach hundreds of thousands of IOPS at low latency, the CPU cost of issuing I/O becomes the bottleneck. The measured characterisation of the main APIs is instructive: at low I/O load all of the APIs perform competitively, and the difference grows with load — with polling contributing a large share of the improvement at the high end. That is the shape of the curve to remember: io_uring does not make a lightly loaded system faster, it removes a ceiling that appears when you push one.

The mechanism in one paragraph

An io_uring instance is a pair of ring buffers shared between kernel and application: the submission queue holds requests, the completion queue holds results, and a memory-mapped header holds the indices. An application writes a submission entry, advances the tail, and — if the kernel is polling the ring (SQPOLL) — never enters the kernel at all. With IOPOLL, completions are polled from the device rather than delivered by interrupt, which removes interrupt overhead on very fast devices at the cost of burning a core. Both modes are per-ring, so you can choose them per workload rather than for the machine.

Measuring the difference on your own hardware

# fio with libaio
fio --name=randread-libaio --ioengine=libaio --direct=1 --rw=randread \
    --bs=4k --iodepth=64 --numjobs=4 --size=8G --time_based --runtime=60 \
    --filename=/dev/nvme0n1 --group_reporting

# fio with io_uring (and with polling)
fio --name=randread-uring --ioengine=io_uring --direct=1 --rw=randread \
    --bs=4k --iodepth=64 --numjobs=4 --size=8G --time_based --runtime=60 \
    --filename=/dev/nvme0n1 --group_reporting

fio --name=randread-uring-iopoll --ioengine=io_uring --iodepth=128 \
    --rw=randread --bs=4k --direct=1 --hipri --numjobs=1 --runtime=60 \
    --filename=/dev/nvme0n1

Compare three numbers per run: IOPS, average latency and CPU utilisation. A kworker count that collapses after switching to IOPOLL is the visible signature of the improvement; if IOPS are identical and CPU time is identical, the workload is not deep enough for the ring to matter, and the migration buys nothing.

Adoption and the practical caveats

What matters operationally is that the API is now mainstream: databases, key-value stores and web servers have adopted it where their workloads are I/O-bound, and modern kernels ship it enabled by default. The caveats are the interesting part. First, polling modes are a core-for-latency trade: hipri on a container with a small CPU quota will reduce throughput rather than improve it, so pin the workload and give it a whole core. Second, kernel version matters — features and security fixes are still landing across releases, so use a recent LTS rather than whatever the distribution shipped three years ago. Third, an io_uring-based application still has to be tuned for the device: iodepth, queue count and NUMA placement dominate the result far more than the choice between io_uring and libaio when the queues are shallow.

When not to bother

If the storage path is a network protocol — NVMe/TCP, iSCSI, NFS — the transport adds latency and CPU cost that dwarfs the syscall overhead io_uring removes, and the right optimisation is a faster transport or a larger queue instead. If the workload is bandwidth-bound rather than IOPS-bound, the syscall rate is already low. And if the application library underneath does not expose the API, migrating the interface without changing the application's queueing structure yields nothing measurable — which is a good reason to benchmark before rewriting anything.

A practical benchmark that answers the right question

Comparing engines in isolation produces numbers nobody can act on. What you want to know is whether your workload of interest moves, and by how much, on your hardware. Build the comparison around three fixed variables — the device, the access pattern, and the queue depth — and vary only the engine.

#!/bin/bash
# engine comparison at a fixed queue depth; run against the same device
DEV=${1:-/dev/nvme0n1}
for ENG in libaio io_uring; do
  for QD in 1 16 64 128; do
    echo "=== engine=$ENG qd=$QD ==="
    fio --name=test --ioengine=$ENG --direct=1 --rw=randread --bs=4k \
        --iodepth=$QD --numjobs=1 --size=8G --time_based --runtime=30 \
        --filename=$DEV --output-format=json \
      | jq -r '.jobs[0] | "IOPS=\(.read.iops) lat_us=\(.read.lat_ns.mean/1000) cpu=\(.usr_cpu + .sys_cpu)"'
  done
done

Read the results as three columns: IOPS, mean latency, and CPU percentage. The pattern to look for is a divergence that grows with queue depth — identical numbers at QD 1, a growing gap at QD 64 and above. If the gap never appears, the store is the bottleneck (or the device is saturated), and the engine choice is irrelevant for that workload. Keep the raw JSON: a benchmark you cannot re-analyse later is a screenshot, not evidence.

Kernel and configuration prerequisites

# kernel version and feature availability
uname -r
grep -i io_uring /boot/config-$(uname -r) 2>/dev/null || zcat /proc/config.gz | grep -i io_uring

# check the security posture of the ring on your kernel
sysctl kernel.io_uring_disabled 2>/dev/null

# per-device queue depth and scheduler sanity
cat /sys/block/nvme0n1/queue/nr_requests
cat /sys/block/nvme0n1/queue/scheduler
nvme id-ctrl /dev/nvme0n1 | grep -i "mdts\|maxcmd"

Three configuration details matter more than most people expect. The I/O scheduler should be none for NVMe, since the device has its own deep queues and the kernel scheduler adds latency rather than reordering benefit. nr_requests should be sized for the queue depths you intend to use. And access to io_uring may be restricted on hardened kernels through kernel.io_uring_disabled — check this before concluding that an application is not using the API, since a blocked ring produces a fallback path with different performance characteristics and no obvious error.

The polling trade and containerised workloads

Polling modes remove interrupt overhead at the price of a spinning core. On bare metal with spare cores, that is a straightforward trade: the core that would have been spent on interrupts spends its time polling instead, and you gain both throughput and latency stability. In a container with a CPU quota, the trade inverts — a spinning core consumes its quota, the scheduler throttles the container, and throughput drops. If you plan to run polling in Kubernetes, the pod needs a Guaranteed QoS class with exclusive CPUs and a full core available to poll.

# a pod that can actually benefit from polling
resources:
  requests: {cpu: "2", memory: "4Gi"}
  limits:   {cpu: "2", memory: "4Gi"}
# plus static CPU manager policy on the kubelet:
#   --cpu-manager-policy=static --reserved-cpus=0-1 --topology-manager-policy=single-numa-node

NUMA placement is the second container consideration: a ring bound to a CPU on one socket, with memory and the device on another, pays a cross-socket penalty on every completion. Modern orchestrators try to place these together, but verify from inside the workload rather than assuming the scheduler got it right on a busy node.

Migration notes for application developers

If you are the one changing the application rather than tuning a system, the pattern that matters most is batching. io_uring rewards submission of many operations per ring flush and punishes one-operation-at-a-time usage, where the setup cost outweighs the benefit. Structure the I/O path so that all operations that can be in flight at once are made in flight — a reactor returning completions to a queue that the application drains in bulk. The second pattern is separation: keep the ring per thread where possible, so that the submission and completion paths share a CPU cache and you avoid cross-thread synchronisation on every operation. Both changes are architectural, which is why the honest advice for a legacy application is usually "adopt the library that uses io_uring" rather than "add io_uring calls".

FAQ

Is io_uring stable enough for production? It has been in the mainline kernel for years and is used by production software, but the details — feature sets, security hardening, behaviour changes — move across releases. Use a recent LTS kernel and pin your version deliberately rather than taking whatever the distribution ships by default.

Does libaio become obsolete? Not overnight. Existing applications work, and the migration cost for a mid-size codebase is real. The pragmatic position is to adopt io_uring for new work and for the bottlenecks that justify the change, not to rewrite working storage paths for a benchmark delta.

What about SPDK? Different trade entirely: a userspace driver that bypasses the kernel for the lowest possible latency, with whole-device ownership and a much larger integration burden. Choose it when you are building a storage product, not when you are optimising an application.

How do I know my application is even using it? Check its documentation and, better, check with a profiler or by observing CPU cost at a fixed I/O rate. An application that reports using io_uring while showing the same syscall profile as before is falling back to a compatibility path.

Related Reading

Related reading: NVMe over TCP: Target and Initiator Setup Guide, Linux NIC Ring Buffer and Coalescing Tuning, NVMe-oF: Choosing Between TCP and RoCEv2

原文链接：https://dl.acm.org/doi/10.1145/3578353.3589545