Elasticsearch and OpenSearch Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. OpenSearch is a fork of Elasticsearch 7.10, created in 2021 after Elastic changed the licence, now maintained under the OpenSearch Software Foundation and licensed Apache-2.0.

The architecture, core APIs and operational model are the same. Unless a page says otherwise, everything here applies to both. See Elasticsearch vs OpenSearch for the differences that have accumulated.

Architecture

• Index — a logical collection of documents, split into shards.
• Shard — a self-contained Lucene index. This is the unit of distribution and the unit of parallelism.
• Replica — a copy of a shard, providing redundancy and read capacity.
• Node roles — master-eligible (cluster state), data (storage and search), ingest, coordinating.
• Segments — immutable Lucene files within a shard, merged in the background.

Documents are analysed into terms at index time and stored in an inverted index, which is what makes relevance-ranked search possible.

Best use cases

• Full-text search with relevance ranking, stemming, synonyms and highlighting.
• Log and observability search over large volumes with time-based indices.
• Faceted navigation, where aggregations over filtered result sets are the query.
• Ad-hoc exploration of semi-structured documents.

When not to use it

• As a primary datastore. See above.
• For frequently updated documents. An update is a delete plus a reindex of the whole document, and deleted documents occupy space until segments merge.
• For relational queries. Joins are limited to specific document relationships and are expensive.
• For very high-cardinality aggregations on huge datasets, where a column store is a better tool.

Near-real-time behaviour

Indexed documents are not immediately searchable. They become visible after a refresh, by default every second per index.

PUT /orders/_settings
{ "index": { "refresh_interval": "30s" } }

Raising the interval substantially improves bulk indexing throughput, at the cost of documents taking longer to appear. For log ingestion, 30 seconds is often a better trade than one second.

Consistency and durability

Writes go to the primary shard and are replicated to replicas. Each write is appended to a translog, flushed to disk according to index.translog.durability — request (default) fsyncs on every request; async is faster and can lose recent writes on a crash.

Common mistakes