Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Elasticsearch and OpenSearchintermediate

Search Cluster Monitoring

Cluster health, the cat APIs, and the metrics that predict a yellow or red cluster.

2 min readIntermediateUpdated Edit this page
ElasticsearchOpenSearch

Cluster health

GET /_cluster/health
GET /_cluster/health?level=indices
GET /_cat/health?v
StatusMeaning
greenAll primary and replica shards allocated
yellowAll primaries allocated, some replicas are not
redAt least one primary shard is unallocated — data is unavailable

Yellow is a redundancy problem; red is a data availability problem. A single-node cluster with replicas configured is permanently yellow by design, which is why an alert should distinguish "unexpectedly yellow" from "yellow as configured". See the red cluster playbook.

GET /_cluster/allocation/explain

This explains exactly why a specific shard is unallocated — the single most useful diagnostic for yellow and red states.

The cat APIs

GET /_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,load_1m,disk.avail
GET /_cat/indices?v&s=store.size:desc
GET /_cat/shards?v&s=state,store:desc
GET /_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET /_cat/pending_tasks?v

rejected in the thread pool output is the metric to watch: rejections mean requests were dropped because the queue was full. Any sustained non-zero rejection count is work the cluster did not do.

Metrics that matter

JVM heap used percent

Source: _nodes/stats/jvm

Sustained above ~75% means memory pressure. Usually caused by shard count, aggregation cardinality or field data.

Old-generation GC frequency and duration

Source: _nodes/stats/jvm

Frequent long pauses destabilise the cluster: a paused node looks unresponsive to the master.

Search and index thread pool rejections

Source: _cat/thread_pool

Requests dropped because queues were full — a direct measure of overload.

Unassigned shards

Source: _cluster/health

Any non-zero value outside a planned operation means reduced redundancy or lost availability.

Disk available per node

Source: _cat/allocation

Crossing the flood-stage watermark makes indices read-only, which requires manual recovery.

Indexing and search latency

Source: _nodes/stats/indices

Per-node index and query time. Rising latency with flat throughput points at merges, GC or slow storage.

Slow logs

PUT /orders/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "1s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "5s"
}

Search slow logs separate the query phase from the fetch phase, which distinguishes an expensive query from expensive document retrieval — two different problems with two different fixes.

Alerts worth having

  • Cluster status red, or yellow when it should be green.
  • Unassigned shards outside a planned operation.
  • JVM heap above threshold, or old-generation GC time rising.
  • Thread pool rejections of any kind.
  • Disk usage approaching the high watermark, with a projection.
  • Snapshot policy failures, or the newest successful snapshot being too old.
  • Lifecycle policy errors, and indices stuck in a phase.
  • Master node changes — frequent elections indicate instability.