Explain Elasticsearch & the Elastic Stack in Observability

1 / 4

Elastic (ELK)

Elasticsearch & the Elastic Stack

Understand the distributed search engine at the core of the ELK stack.

Elasticsearch is a distributed, document-oriented search and analytics engine built on Apache Lucene, and it is the heart of the Elastic (ELK) Stack. Data is stored as JSON documents in indices; each index is split into shards (primary and replica) spread across nodes, which is how Elasticsearch scales horizontally and stays available when a node fails. The defining feature is the inverted index: every term is mapped to the documents containing it, making full-text search and aggregations extremely fast.

The index at the back of thousands of books merged into one: instead of reading every page, you look up a word and instantly get every page it appears on, across the whole library.

Key Concepts

The classic stack is Elasticsearch + Logstash + Kibana ("ELK"), later "Elastic Stack" once Beats joined. Elasticsearch does the storage, search, and aggregation; the others handle ingestion and visualization. Unlike Loki's label-only indexing, Elasticsearch indexes content richly, which makes arbitrary full-text queries and analytics powerful — at the cost of more storage and compute. A mapping defines how each field is indexed, and getting mappings right (text vs keyword, avoiding field explosions) is central to performance.

What the interviewer is probing.

What the interviewer is probing.

1. "What is the stack made of?" *Probing:* the vocabulary. *Stalls:* "Elasticsearch." *Moves up:* Elasticsearch for storage and search, Kibana for the UI, plus Beats and Logstash for ingestion.

1. "What is the stack made of?"

2. "What is an index and a shard?" *Probing:* the fundamentals. *Stalls:* "Storage." *Moves up:* an index is a logical collection split into shards, each a Lucene index — shard count sets parallelism and cannot be changed after creation.

2. "What is an index and a shard?"

3. "What is the oversharding problem?" *Probing:* the operational trap. *Stalls:* "None." *Moves up:* many small shards each carry fixed overhead, so thousands of them exhaust heap and slow the cluster.

3. "What is the oversharding problem?"

4. "What is ILM for?" *Probing:* the lifecycle. *Stalls:* "Nothing." *Moves up:* index lifecycle management — rolling over, moving to cheaper tiers and deleting on a policy rather than by hand.

4. "What is ILM for?"