Cluster Sizing

Sizing decisions here are unusually consequential because primary shard count cannot be changed after index creation.

Shard sizing

The commonly cited target is tens of gigabytes per shard — often quoted as roughly 10–50 GB. The reasoning behind the range matters more than the numbers:

• Too small — overhead per shard dominates; hundreds of tiny shards consume heap and file handles for no benefit and lengthen every fan-out query's tail.
• Too large — recovery and rebalancing move enormous units, and a single shard's merge work becomes disruptive.

Estimate the primary count from expected index size divided by the target shard size, then round to something that divides evenly across your data nodes.

GET /_cat/shards?v&s=store:desc
GET /_cat/allocation?v
GET /_cluster/stats?filter_path=indices.shards.total,indices.store.size_in_bytes

Heap Management

# jvm.options
-Xms31g
-Xmx31g

Minimum and maximum must be identical, so the JVM does not resize the heap at runtime.

GET /_nodes/stats/jvm?filter_path=nodes.*.jvm.mem.heap_used_percent,nodes.*.jvm.gc

Sustained heap usage above roughly 75%, or frequent old-generation garbage collections, means the node is under memory pressure. The causes are usually shard count, aggregation cardinality, or field data — in that order.

Node roles

Separating roles keeps a heavy query from destabilising the cluster state:

# Dedicated master-eligible node: small heap, no data.
node.roles: [ master ]
 
# Data node.
node.roles: [ data, ingest ]
 
# Coordinating-only node, useful in front of heavy aggregation traffic.
node.roles: [ ]

Run exactly three master-eligible nodes for quorum. More does not improve availability and makes elections slower.

Circuit breakers

indices.breaker.total.limit: 70%
indices.breaker.request.limit: 60%
indices.breaker.fielddata.limit: 40%

Circuit breakers reject a request that would exceed a memory bound instead of letting it exhaust the heap. A rejected query is far better than an out-of-memory node that takes its shards with it — treat breaker trips as a signal to fix the query, not as a reason to raise the limit.

Disk watermarks

cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%

At the flood stage, indices with shards on the affected node are set read-only. Recovering requires freeing space and clearing the read-only block manually:

PUT /_all/_settings
{ "index.blocks.read_only_allow_delete": null }

Knowing that second step exists is what turns a 3am outage into a five-minute fix.