systemd-resolved and resolvectl: DNS Troubleshooting

Modern distributions route DNS through systemd-resolved: applications query a local stub at 127.0.0.53, and resolved forwards those queries per link. When it works, nobody notices. When it breaks you get three different failure modes — nothing resolves, only short names fail, or one application works while another does not — and the usual reflex of editing /etc/resolv.conf does nothing because the file is a symlink that gets regenerated. Here is the mental model and the command set that actually diagnoses it.

Which Mode Is resolved Running In?

systemd-resolved has three distinct modes and the symptoms differ in each, so identify the mode before changing anything. The mode is decided entirely by what /etc/resolv.conf points at, and getting this wrong is why so many \"DNS fixes\" appear to work and then evaporate on reboot.

ls -l /etc/resolv.conf
resolvectl status | head -20
systemd-resolve --status 2>/dev/null | head -5   # legacy name, older releases
cat /run/systemd/resolve/stub-resolv.conf

If /etc/resolv.conf is a regular file rather than a symlink, something else owns DNS on this host — a container runtime, a VPN client, or an administrator who edited it directly. On containers this is the normal state and resolved is usually not running at all; check systemctl is-active systemd-resolved before assuming resolved is the culprit.

How the Pieces Fit

• Stub listener: resolved listens on 127.0.0.53:53 (not 127.0.0.1). Applications pointing at it get caching, per-link DNS and DNSSEC validation.
• /etc/resolv.conf: in the recommended "stub mode" it is a symlink to /run/systemd/resolve/stub-resolv.conf, containing only nameserver 127.0.0.53 plus search domains.
• NSS: the resolve module in /etc/nsswitch.conf lets glibc call resolved directly over D-Bus.
• Per-link configuration: NetworkManager, systemd-networkd or DHCP provides the DNS servers per interface; resolved merges them with route-only domains.

# the canonical stub-mode symlink
ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

First Five Commands

resolvectl status                 # global + per-link DNS servers, DNSSEC, LLMNR/mDNS state
resolvectl query example.com      # the query resolved would actually send, with the link used
resolvectl domain                 # search and route-only (~) domains per link
resolvectl dns                    # DNS servers per link
resolvectl statistics             # cache hits/misses, failures — cache size often matters

If a hostname resolves through resolvectl query but not from an application, the problem is the NSS path or a sandboxed service with its own resolv.conf, not the DNS servers.

Per-Link DNS and Route-Only Domains

resolved keeps a separate DNS server list, search domain list and DNSSEC setting per network interface, and merges them per query. Two pieces of syntax do most of the work:

• A plain domain (corp.example.com) is a search domain: it is appended to short names, and queries for it go to that link's servers first.
• A domain prefixed with a tilde (~corp.example.com) is route-only: it is never appended to short names, but any query for that suffix is sent only to that link's servers. This is how you keep internal names off public resolvers.

resolvectl domain                        # what each link currently has
sudo resolvectl domain eth0 '~corp.example.com' 'corp.example.com'
sudo resolvectl domain wg0 '~corp.example.com'   # VPN keeps its own split DNS

# persistent equivalent in a drop-in
sudo resolvectl dns eth0 10.10.10.1 10.10.10.2
sudo resolvectl dnsovertls eth0 yes      # if the upstream supports DoT
sudo resolvectl flush-caches

The tilde matters more than it looks. A VPN that pushes corp.example.com as a plain search domain will also send queries for that suffix to whatever public resolver is first in the merged list when the VPN is down, which leaks internal hostnames — and, on a hostile network, hands an attacker a list of names to spoof. Route-only domains close that hole.

DNSSEC and DNS over TLS

DNSSEC validation turns a wrong answer into an error instead of a silent redirect. That is the point, and it is also the source of most \"some sites do not load\" reports on resolved hosts: a domain with a broken chain or a resolver that strips DNSSEC records produces SERVFAIL for that domain alone, while everything else resolves normally.

resolvectl query --validate example.com      # show the validation result explicitly
resolvectl status | grep -i dnssec
resolvectl statistics | grep -i fail

The setting most deployments should use is DNSSEC=allow-downgrade: validate when the upstream supports it, fall back when it does not. yes makes validation mandatory and will break on captive portals and on any upstream that filters DNSSEC. Turning DNSSEC off entirely (no) to \"fix\" one broken domain is the wrong trade — record which domain failed and report it instead. The same discipline applies to DNSOverTLS: enable it only for upstream servers that actually speak DoT, because the strict setting makes every query to a plain resolver fail.

Common Failures and Their Fixes

Persistent Configuration

sudo mkdir -p /etc/systemd/resolved.conf.d
sudo tee /etc/systemd/resolved.conf.d/99-dns.conf <<'EOF'
[Resolve]
DNS=10.10.10.1 10.10.10.2
FallbackDNS=1.1.1.1
Domains=~corp.example.com
LLMNR=false
DNSSEC=allow-downgrade
EOF

sudo systemctl restart systemd-resolved
resolvectl flush-caches            # after changing upstreams or DNS records
systemctl status systemd-resolved  # confirm it is the one on 127.0.0.53:53

lsof -i @127.0.0.53:53 proves the stub is running and who owns the port; dig @127.0.0.53 example.com bypasses NSS and tests resolved alone. Comparing that with dig @8.8.8.8 example.com isolates "resolved is broken" from "the network cannot reach DNS".

A Diagnostic Path That Converges

DNS complaints are usually one of three things: resolved is misconfigured, the network cannot reach the configured server, or the application is not using the stub at all. Test in that order and stop at the first failure.

1. Is resolved alive and owning the stub port? systemctl status systemd-resolved and lsof -i @127.0.0.53:53.
2. Does resolved itself resolve? resolvectl query example.com, which reports the link and server it used. If this fails, the problem is upstream configuration.
3. Does a direct query to resolved work, bypassing NSS? dig @127.0.0.53 example.com. If this works but resolvectl query does not, NSS ordering is wrong.
4. Can the network reach a public resolver? dig @1.1.1.1 example.com. If this fails, DNS is a symptom of a routing or firewall problem, not a resolver problem.
5. Does the application see the same servers? Containers, chroots and systemd services with PrivateNetwork= often carry their own /etc/resolv.conf.

Recording the output of resolvectl status before and after each change turns a vague \"DNS is broken\" ticket into a two-line diff. The commands below are the ones worth keeping in a runbook.

Operations Habits

1. Treat resolved as configuration-managed: keep DNS settings in resolved.conf.d or in network profiles, never hand-edited in resolv.conf.
2. Log DNS server changes with the interface state — most "random" resolution failures correlate with a link flap or VPN connect.
3. Give internal domains route-only (~) assignments so they never leak to public resolvers.
4. After editing upstream servers, always flush the cache; stale negative entries cause confusing "it works for me, not for you" reports.

Worked Case: Only the Internal Zone Fails

A host resolves the public internet perfectly but cannot reach git.corp.example.com. resolvectl query git.corp.example.com returns NXDOMAIN, and resolvectl status shows the corporate server is configured on the eth0 link but the domain list holds only the DHCP-provided search domain. The query is therefore treated as a public name and sent to the fallback resolver, which correctly answers NXDOMAIN. Assigning the suffix as route-only on the correct link fixes it immediately and permanently:

sudo resolvectl domain eth0 '~corp.example.com'
resolvectl query git.corp.example.com

The general lesson is that \"DNS works\" and \"this domain resolves\" are different statements. Whenever exactly one zone fails while everything else resolves, the question to ask is not which server is configured but which link the query was routed to — resolvectl query prints exactly that.

Housekeeping That Prevents Surprises

• Cap the cache if the host churns through many distinct names: CacheSize= and CacheMaxTTL= bound memory and stop very long TTLs from pinning stale records.
• Set DNSStubListener= deliberately. On hosts running a local Dnsmasq, Pi-hole or Unbound, two listeners cannot share 127.0.0.53:53 — move one of them to another address rather than disabling resolved wholesale.
• Keep a fallback resolver in resolved.conf. DHCP that fails to deliver DNS, or a router reboot, leaves the host with no servers and a conspicuously silent resolution path.
• Document the internal suffixes with their ~ markers in configuration management, so a rebuilt host inherits split-DNS behaviour instead of leaking internal names to public resolvers.

相关阅读：Linux 网络内核参数调优、OpenSSL 证书链校验排障 以及 NGINX 反向代理配置。

原文链接：ArchWiki - systemd-resolved