Docker overlay2 Disk Full: Safe Cleanup Order

write /var/lib/docker/overlay2/...: no space left on device usually appears before df -h looks alarming, because Docker keeps its own directory tree and a few containers can fill an entire partition. The recovery is a sequence, not a single command: find out whether you are out of bytes or inodes, find what actually owns the space, reclaim in order of safety, and then stop the host from refilling. This guide follows that order and flags the one command that can destroy data.

Step 1: bytes or inodes?

df -h /var/lib/docker
df -i /var/lib/docker
du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head

Both exhaustion modes produce the identical error message. If df -i shows 100% inodes, deleting large files will not help; you need to remove many small files (typically a huge number of tiny layers or log files).

Step 2: find the real consumer

docker system df -v
sudo du -h /var/lib/docker/containers/*/*-json.log | sort -rh | head
sudo du -sh /var/lib/docker/overlay2/* | sort -rh | head -5

On a healthy host, /var/lib/docker/containers holds tens of megabytes. When a container in a crash loop writes stack traces, the default json-file logging driver has no size limit at all and can fill the disk on its own. If a single overlay2 hashed directory dominates, map it to its container before deleting anything – that directory may be the running container's writable layer.

Step 3: reclaim in order of safety

# safe: stopped containers and dangling images only
docker container prune -f
docker image prune -a -f
docker network prune -f

# build cache (excluded from docker system prune on newer engines)
docker builder prune -a -f

# volumes: DATA LOSS, verify first
docker volume ls
docker volume prune -f

# everything unused in one pass
docker system prune -a --volumes -f
docker system df     # confirm what was reclaimed

Typical breakdown on a build host is build cache first, images second, volumes and stopped containers after that. Never run the volume variant on a host that stores databases or uploads in named volumes – a volume prune deletes unreferenced volumes, which is exactly the case when a stack is temporarily down.

Truncating a runaway log without breaking the daemon

sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log

Use truncate, not rm: the daemon holds an open file descriptor and removing the file leaves the space allocated until the process restarts.

Step 4: stop it refilling

# /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

Apply it with a daemon restart, and add a bounded periodic prune:

# /etc/cron.daily/docker-prune
#!/bin/sh
docker image prune -af --filter "until=168h" >/dev/null 2>&1
docker builder prune -af --filter "until=168h" >/dev/null 2>&1

Step 5: relocate the data root if the partition is simply too small

# /etc/docker/daemon.json
{ "data-root": "/srv/docker" }

systemctl stop docker
rsync -aHAX /var/lib/docker/ /srv/docker/
systemctl start docker
docker info | grep "Docker Root Dir"

Stop the daemon before copying, preserve extended attributes and hardlinks, and verify the new root before deleting the old directory. On cloud instances, resizing the attached volume is usually simpler and safer than relocating the data root.

Monitoring

• Alert on filesystem usage above 80% and on inode usage, not just on bytes.
• Track docker system df growth over time; build cache growing linearly is a CI configuration problem, not a Docker one.
• Log rotation is per-container when the driver options are set at the daemon level – check that any container started with an explicit --log-driver also has limits.

Related: Kubernetes Gateway API vs Ingress, LVM thin provisioning, and smartctl disk health monitoring.

原文链接：https://stackharbor.com/en/knowledge-base/docker-overlay2-disk-full-cleanup