Elasticsearch Red/Yellow: Unassigned Shards Fix

Yellow means replicas are missing; red means primary shards are missing and some data is unsearchable. Both are allocation problems, not data-loss problems, and the allocation explain API tells you exactly which rule is blocking recovery. This guide covers the diagnostic sequence, the usual root causes, and the fixes that do not create more risk than the incident.

Read the cluster state properly

GET _cluster/health?filter_path=status,unassigned_shards,number_of_nodes,active_primary_shards
GET _cat/indices?v&health=red
GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state
GET _cluster/allocation/explain

Start with health, narrow to the unhealthy indices, list the unassigned shards and then ask why. The explain call returns can_allocate with a reason such as no_valid_shard_copy, disk_threshold, max_retries or no_attempt, which is the difference between guessing and acting.

Common causes, in the order worth checking

1. Node loss

Check for repeated node-leaving messages in the logs. When the node returns, Elasticsearch reallocates automatically but delays allocation by one minute by default to avoid churn from transient blips. Before the node comes back, confirm the shards are not held by an allocation setting that pins them to the missing node.

2. Disk watermarks

GET _cat/allocation?v
GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk*

The low/high/flood-stage watermarks (typically 85/90/95%) stop allocation and then move shards away. Raising a watermark is a temporary measure – it buys time to add capacity or delete indices, and it should never be left in place. A cluster can be genuinely red because no data node has room for the primary.

3. Replica count higher than the number of nodes

A single-node cluster is permanently yellow: replicas can never be assigned to the node that already holds the primary. Fix it deliberately:

PUT /index-name/_settings
{ "index": { "number_of_replicas": 0 } }

4. Allocation filters and shard limits

PUT /index-name/_settings
{ "index.routing.allocation.require._name": null, "index.routing.allocation.include._name": null }
GET _cat/allocation?v&h=node,shards,disk.percent

Leftover index.routing.allocation settings from a migration are a frequent culprit, as is cluster.max_shards_per_node on clusters that have grown a lot of small daily indices.

Fixes that carry risk

• Red with primaries lost for good – the only paths are restoring the index from a snapshot or, if the data is expendable, deleting the index. Do not force-allocate an unassigned primary onto a node that lacks the shard data: you will recover an empty shard and lose the real one.
• restore from snapshot – choose the newest snapshot that contains the index, and consider restoring under a temporary name to validate before swapping.
• reroute API – POST _cluster/reroute moves or allocates shards manually; treat it as a last resort and record what you did.

Monitoring so it does not recur

• Alert on status transitions and on unassigned_shards > 0 for more than a few minutes, not on red only.
• Watch disk usage per node, not cluster-wide; allocation is decided per node.
• Use ILM so old indices shrink/roll over instead of accumulating shards, and keep the number of shards per node sane – thousands of tiny shards cause slower recovery than the red status ever would.
• Verify snapshots restore, on a schedule, not just that they completed.

Related: Grafana Loki log aggregation, Ceph OSD and PG triage, and ZFS send/receive replication.

原文链接：https://www.elastic.co/docs/troubleshoot/monitoring/unavailable-shards