node-problem-detector: Node Health the Cluster Can See

Kubernetes knows a node is unhealthy when the kubelet stops posting status — which covers crashes and network partitions, and nothing else. It does not know that the node's kernel is logging an OOM cascade, that a disk has developed read errors, or that the container runtime is being restarted repeatedly. node-problem-detector (NPD) fills that gap: a small daemon per node that watches system logs and configurable checks, then publishes what it finds as node events and conditions the rest of the cluster can react to. This guide covers deployment and the parts that make it genuinely useful rather than a log shipper with a better name.

Two kinds of monitor

Each configured monitor runs as its own goroutine inside the daemon, and each has its own rule set, so you can run one monitor for the kernel log and another for a runtime log without them interfering. Rules match log patterns and map them to a condition with a status, a reason and a message — which is the payload everything downstream consumes.

Deployment

# fastest path: the Helm chart
helm repo add deliveryhero https://charts.deliveryhero.io/
helm install node-problem-detector deliveryhero/node-problem-detector

# manual: a DaemonSet with the system log directory mounted
kubectl create -f node-problem-detector-config.yaml   # ConfigMap with monitor configs
kubectl create -f node-problem-detector.yaml          # DaemonSet

kubectl -n kube-system get pods -l app=node-problem-detector
kubectl get events --field-selector reason=OOMKilling -A | head
kubectl describe node node-1 | sed -n '/Conditions:/,/Addresses:/p'

Key flags when running standalone or tuning the DaemonSet: --system-log-monitors takes a comma-separated list of monitor configuration files, --custom-plugin-monitors the same for plugin checks, and --address/--port control the local server. Mount the host's /var/log (or the specific log directory) — a container that cannot see the kernel log can only report on itself, which is useless.

Writing a useful plugin check

{
  "plugin": "custom",
  "pluginConfig": {
    "invoke_interval": "60s",
    "timeout": "20s",
    "max_output_length": 200,
    "concurrency": 1
  },
  "source": "custom-plugin-monitor",
  "conditions": [
    {
      "type": "ReadonlyFilesystem",
      "reason": "FilesystemReadonly",
      "message": "root filesystem is mounted read-only"
    }
  ],
  "rules": [
    {
      "type": "temporary",
      "reason": "FilesystemUnhealthy",
      "pattern": "READONLY"
    }
  ]
}

Pair it with a shell script that exits non-zero and prints the pattern when the problem is present. Good candidates for plugin checks are the ones your on-call engineer would otherwise discover by SSH: a filesystem that remounted read-only, a NIC that dropped to a lower link speed, an NTP offset beyond tolerance, or a hardware sensor warning from ipmitool. Keep the check fast and side-effect free; NPD runs it on every node on an interval, so a script that pages through a huge log file becomes its own problem.

Reacting to what it finds

# a node condition shows up as standard Kubernetes condition
kubectl get node node-1 -o jsonpath='{.status.conditions[?(@.type=="KernelDeadlock")].status}'

# Prometheus can scrape NPD's metrics endpoint and alert on condition churn
# node_problem_detector_problem_counter{condition="ReadonlyFilesystem"} > 0

# and you can act automatically: cordon a node whose problems are hardware-like
kubectl cordon node-1

The value comes from wiring the signal into something. At minimum, alert on problem counters (especially the ones that should never occur, such as readonly filesystem or kernel deadlock) and include the node condition in your triage dashboard. If you run automation, cordoning a node that reports a hardware problem is a defensible automatic action — better to drain than to keep scheduling pods onto failing hardware. What you should not do is treat every NPD event as a page: the default ruleset is intentionally chatty about transient conditions, and alert fatigue is the fastest way to make this tool worthless.

Designing the rule set: what deserves a condition

The default ruleset watches a broad set of kernel and runtime messages, and the temptation is to enable everything and alert on all of it. That produces noise, and noise is the failure mode that makes node health monitoring worthless. The design question is which findings should become a Kubernetes condition, which should become an alert, and which should only be logged.

The principle: a condition should mean "a human may need to act on this node", and everything else belongs in metrics. Conditions churn, and clients that watch them — autoscalers, operators, your own automation — react to changes, so a condition that flips on and off is actively harmful.

From condition to action: wiring it into the cluster

# 1) metrics: scrape NPD and alert on problem counters
# node_problem_detector_problem_counter{condition="ReadonlyFilesystem"} > 0
# node_problem_detector_problem_gauge{condition="KernelDeadlock"} > 0

# 2) automation that reacts to the noisy-but-real cases
#    a small controller or a CronJob that cordons nodes with hardware-like conditions
kubectl get nodes -o json | jq -r '
  .items[] | select(.status.conditions[]? | select(.type=="KernelDeadlock" and .status=="True")) | .metadata.name'
kubectl cordon node-1

# 3) surface conditions in the triage view, next to the usual suspects
kubectl describe node node-1 | sed -n "/Conditions:/,/Addresses:/p"
kubectl get events -A --field-selector reason=NodeProblem --sort-by=.lastTimestamp | tail

Cordoning automatically is a defensible action for hardware-shaped problems, because keeping pods off a failing node is nearly always right. Evicting automatically is a much bigger step: eviction of a node that is merely flapping causes an availability incident of its own. Draw that line explicitly, document it in the runbook, and make it a property of the condition type rather than of whoever happens to be on call.

Custom checks worth writing

The plugin monitor is where NPD earns its place beyond a fancy log reader. Four checks repay the effort on almost any fleet:

# 1) read-only filesystem, or an expected mount missing
mount | grep -q " / " || echo "ROOT_MOUNT_MISSING"
findmnt -no OPTIONS / | grep -qw ro && echo "READONLY"

# 2) NIC link speed dropped from the expected value
ethtool eth0 | grep -q "Speed: 25000Mb/s" || echo "LINK_SPEED_DEGRADED"

# 3) time offset against a trusted source
chronyc tracking | awk '/System time/ {print $4}' | awk '{ if ($1 > 0.5) print "NTP_OFFSET_HIGH" }'

# 4) hardware sensor warning via IPMI where available
ipmitool sdr type Temperature 2>/dev/null | grep -i "upper critical" && echo "THERMAL_WARNING"

Set the invoke interval in proportion to the cost of the check — a check that shells out to IPMI should run every few minutes, not every thirty seconds — and keep the output short, since it becomes an event message that a human will read in a terminal. Each of these checks catches a class of problem that otherwise surfaces as a vague service degradation.

FAQ

Does NPD replace node monitoring (node_exporter and friends)? No — it complements it. node_exporter gives you resource metrics; NPD gives you interpreted problem signals as Kubernetes objects. Run both, and alert on NPD's conditions alongside your metric-based alerts.

How much resource does it use? It is a small daemon whose cost is dominated by how often your custom checks run and how much log they read. Keep checks cheap and intervals sane, and measure rather than assuming.

Where do the events go? Into the node's event stream and, with metrics enabled, into your monitoring system. Configure an alerting route for the condition set you chose as page-worthy, and leave the rest as metric-only.

Can I use it to detect a problem in a specific application? Only if the problem surfaces in a log or a command you can check. For application-level health, a liveness probe or an application-specific metric is the better tool; NPD's scope is the node.

Quick reference: deploy, verify, react

# deploy via Helm (fastest) or as a DaemonSet with a ConfigMap of monitor configs
helm install node-problem-detector deliveryhero/node-problem-detector

# verify it is running on every node and reading the right logs
kubectl -n kube-system get pods -l app=node-problem-detector -o wide
kubectl -n kube-system logs -l app=node-problem-detector --tail=30

# see what it has reported
kubectl get events -A --field-selector reason=NodeProblem
kubectl describe node node-1 | sed -n '/Conditions:/,/Addresses:/p'

# scrape its metrics and alert on the condition set you chose as page-worthy
kubectl -n kube-system port-forward svc/node-problem-detector 20257:20257
curl -s localhost:20257/metrics | grep node_problem_detector

Verify on more than one node before trusting the deployment: a DaemonSet that is running everywhere but mounted the wrong log path reports nothing at all, which looks identical to a healthy cluster. Force a known problem in a test cluster — remount a filesystem read-only, or trip one of your custom checks — and confirm the condition appears, the event is emitted and the metric increments. A monitoring component that has never been seen to fire is a component you cannot rely on during an incident.

Where NPD fits in the triage flow

The useful mental model is that NPD owns the node's own opinion of its health, while metrics own the node's resource consumption and logs own the detailed narrative. In a triage sequence, a service-level alert tells you something is broken, the node conditions tell you whether the fault is localised to a node or spread across the cluster, and the logs then explain why. Without the middle step, every incident starts by manually checking each node for the obvious hardware and kernel problems, which is exactly the work a five-minute DaemonSet deployment removes.

Keep the coverage honest in the other direction too: NPD cannot see a failing disk controller that never logs an error, and it cannot see a network path degrading on the far side of a switch. It narrows the search space, it does not close it — and stating that limitation in your runbook prevents someone from concluding "node conditions are clean, therefore the node is fine" when the real cause is outside the host.

Related Reading

Related reading: Kubernetes 节点 NotReady 排障：从 kubelet 到 CNI, Alertmanager Routing, Grouping and Silences That Work, Kubernetes Autoscaling: HPA vs VPA vs KEDA

原文链接：https://pkg.go.dev/github.com/kubernetes/node-problem-detector