Elasticsearch Internals

Distributed search and analytics engine built on Apache Lucene. Horizontally scalable, schema-flexible, and optimized for full-text search and near-real-time analytics.

1. What Elasticsearch Is

• Search engine: inverted index for full-text search with relevance scoring
• Analytics engine: aggregations over large datasets (like SQL GROUP BY but distributed)
• Document store: JSON documents, schemaless or schema-controlled via mappings
• Built on Lucene: each shard is a Lucene index; ES adds distribution, replication, and a REST API

Not a primary database — no ACID transactions, no joins across indexes. Use it for search, log analytics (ELK stack), and metrics aggregation.

2. Core Architecture

Node Types

A node can have multiple roles. In production, dedicate master and data nodes.

3. Cluster Topology Diagram

graph TD
    classDef client fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef coord fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef masterActive fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef masterStandby fill:#f5b7b1,stroke:#c0392b,color:#1a1a1a
    classDef data fill:#2980b9,stroke:#1f618d,color:#fff
    classDef primary fill:#27ae60,stroke:#1e8449,color:#fff
    classDef replica fill:#58d68d,stroke:#1e8449,color:#1a1a1a
    classDef ingest fill:#f39c12,stroke:#ba6018,color:#fff

    CLIENT["Client / application"]:::client -->|"REST request"| Coord["Coordinating Node<br/>routes + scatters/gathers<br/>(every node does this by default)"]:::coord

    subgraph MASTERS["Master-eligible quorum"]
        MN1["Master Node 1<br/>ACTIVE — owns cluster state"]:::masterActive
        MN2["Master Node 2<br/>standby, votes in elections"]:::masterStandby
        MN3["Master Node 3<br/>standby, votes in elections"]:::masterStandby
        MN1 -.->|"publishes cluster state<br/>(Raft-based since ES 7)"| MN2
        MN1 -.-> MN3
    end

    Coord -.->|"cluster state, shard routing table"| MN1

    subgraph DATA["Data nodes — hold shards, execute queries"]
        DN1["Data Node 1"]:::data
        DN2["Data Node 2"]:::data
        DN3["Data Node 3"]:::data
        DN1 --> P0["Primary Shard 0"]:::primary
        DN1 --> R1["Replica Shard 1"]:::replica
        DN2 --> P1["Primary Shard 1"]:::primary
        DN2 --> R2["Replica Shard 2"]:::replica
        DN3 --> P2["Primary Shard 2"]:::primary
        DN3 --> R0["Replica Shard 0"]:::replica
    end

    Coord --> DN1
    Coord --> DN2
    Coord --> DN3

    ING["Ingest Node<br/>runs pipelines before indexing"]:::ingest -->|"pre-processed doc"| DN1

Note that every replica lives on a different data node than its own primary (R0 is on DN3, not DN1 alongside P0) — that's what lets a replica keep serving reads, and get promoted, if the node holding its primary disappears entirely. The master nodes never touch document data at all; they only own cluster state (which indices exist, which shard lives on which node) and get elected among themselves independently of the data path.

4. Inverted Index

Traditional databases index rows → fields. Lucene inverts this: term → list of document IDs.

Tokenization Example

Input text: "The quick brown fox"

Analysis pipeline:

1. Char filter — strip HTML, map chars (e.g., & → and)
2. Tokenizer — split on whitespace: [The, quick, brown, fox]
3. Token filters — lowercase, stop words, stemming: [quick, brown, fox] (the removed as stop word)

Resulting inverted index:

When you search for "quick fox":

1. Tokenize query → [quick, fox]
2. Look up each term in inverted index
3. Intersect doc ID lists → [1, 4] for quick ∩ [1] for fox = [1]
4. Score by TF-IDF or BM25

The keyword field type skips analysis — stored as-is for exact matching, sorting, and aggregations.

Try It Yourself: Live Document Indexer

A simplified version of the same term → doc IDs mechanism above: tokenization here is just lowercase + split on whitespace + strip basic punctuation (no stop-word removal, no positions, no scoring — real Lucene does all three). Insert a few sentences, search a term, and watch which document boxes and which postings entry light up. Delete removes a doc immediately in this demo; the status message after a delete explains how that differs from real Elasticsearch.

5. Index Segments

A Lucene index (shard) is composed of immutable segments.

How segments work

• Write: documents first go to an in-memory buffer + translog
• Refresh (default every 1s): memory buffer flushes to a new on-disk segment → document becomes searchable
• Flush (triggered by translog size or time): fsync segments + translog to disk, clear translog
• Merge: background merging combines small segments into larger ones, physically deletes documents marked in the delete bitmap

Why writes are near-real-time, not real-time

The refresh operation (buffer → segment) happens every second by default. Documents are not searchable until the next refresh. For immediate visibility, use refresh=true on the index request (expensive — avoid in bulk).

Deleted documents

ES marks deletes in a bitmap (.del file). The document still occupies disk until a merge physically removes it. _forcemerge compacts this.

Segment merge

graph LR
    classDef small fill:#3498db,stroke:#2471a3,color:#fff
    classDef del fill:#c0392b,stroke:#7b241c,color:#fff
    classDef merged fill:#27ae60,stroke:#1e8449,color:#fff

    S1["Segment 1<br/>100 docs<br/>(3 marked deleted)"]:::small --> M["Merge policy<br/>picks candidate segments"]
    S2["Segment 2<br/>80 docs<br/>(0 marked deleted)"]:::small --> M
    S3["Segment 3<br/>40 docs<br/>(7 marked deleted)"]:::small --> M
    DEL["Delete bitmap (.del)<br/>tombstones from all 3 segments"]:::del --> M
    M --> OUT["Merged Segment<br/>170 docs — 10 tombstones<br/>physically purged, disk reclaimed"]:::merged

Merges are CPU/IO intensive. During bulk indexing, set refresh_interval: -1 and number_of_replicas: 0, then restore after.

6. Write Path Sequence Diagram

sequenceDiagram
    participant C as Client
    participant CN as Coordinating Node
    participant PN as Primary Shard
    participant R1 as Replica Shard 1
    participant R2 as Replica Shard 2

    rect rgb(40, 55, 71)
    Note over C,R2: Indexing — write must reach every in-sync replica before ACK
    C->>CN: PUT /index/_doc/1
    CN->>CN: Route via hash(doc_id) % num_shards
    CN->>PN: Forward to primary shard owner
    PN->>PN: Write to translog (durability)
    PN->>PN: Write to in-memory buffer (not yet searchable)
    par replicate in parallel
        PN->>R1: Replicate op
        R1->>R1: Write to translog + buffer
        R1-->>PN: ACK
    and
        PN->>R2: Replicate op
        R2->>R2: Write to translog + buffer
        R2-->>PN: ACK
    end
    PN-->>CN: ACK — wait_for_active_shards satisfied
    CN-->>C: 201 Created
    end

    rect rgb(52, 73, 44)
    Note over PN,R2: Refresh — every refresh_interval (1s default), independently per shard copy
    PN->>PN: Flush buffer to new Lucene segment
    R1->>R1: Flush buffer to new Lucene segment
    R2->>R2: Flush buffer to new Lucene segment
    Note over PN,R2: Document now searchable on every copy — but not yet fsynced to disk
    end

    rect rgb(74, 46, 46)
    Note over PN,R2: Flush — on translog-size/time trigger, per shard copy
    PN->>PN: fsync segments + translog, clear translog
    R1->>R1: fsync segments + translog, clear translog
    R2->>R2: fsync segments + translog, clear translog
    Note over PN,R2: Now durable — survives a crash on every copy
    end

The client only ever gets a 201 after every in-sync replica has acknowledged the write — that's the indexing phase (top block). Refresh and flush happen later, independently on each shard copy, and don't block the client at all: the document is durable in the translog well before it's searchable, and searchable well before its segment is fsynced.

7. Sharding

Shard routing

shard_id = hash(document_id) % number_of_primary_shards

This is why primary shard count is fixed at index creation — changing it would invalidate the routing formula for all existing documents. To resize, use _reindex into a new index with different shard count, or use _split/_shrink API.

Primary vs Replica

• Primary: handles all writes, replicates to replicas
• Replica: serves read requests, promoted to primary if primary fails
• Replicas are never on the same node as their primary (cluster moves them automatically)

Shard sizing rule

• Target 20–50 GB per shard
• Too small: overhead from too many small Lucene indexes
• Too large: slow recovery, rebalancing is expensive
• Rule of thumb: num_shards = total_data_size / 30GB

8. Replication

Write flow (sync)

1. Client writes to primary shard
2. Primary validates and writes locally
3. Primary forwards to all in-sync replica shards in parallel
4. Waits for all replicas to ACK (controlled by wait_for_active_shards)
5. Returns success to client

In-Sync Replicas (ISR)

ES tracks which replicas are "in sync" with the primary. Lagging replicas are removed from ISR. Primary only waits for ISR replicas.

Primary failure

1. Master detects primary is down
2. Promotes an in-sync replica to new primary
3. Assigns a new replica on another node
4. Old primary (if it recovers) is fenced — must re-sync before serving writes

9. Cluster States

What triggers each

Yellow (most common in single-node clusters):

• Only 1 node — no place to put replicas
• Node left the cluster and its replicas are unassigned
• Fix: add nodes, or set number_of_replicas: 0 for dev

Red:

• Node with primary shard(s) is down and no in-sync replica exists
• Corrupt shard data
• Fix: restore from snapshot, or use _cluster/reroute to allocate stale replica

# Check cluster health
GET /_cluster/health

# See unassigned shards
GET /_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

# Explain why a shard is unassigned
GET /_cluster/allocation/explain

10. Mappings

Dynamic vs Explicit

Dynamic mapping: ES auto-detects types on first document. Risky — a string "123" maps to long, next doc with "abc" fails.

Explicit mapping: Define upfront, prevents surprises in production.

PUT /products
{
  "mappings": {
    "properties": {
      "name":        { "type": "text", "analyzer": "english" },
      "sku":         { "type": "keyword" },
      "price":       { "type": "float" },
      "created_at":  { "type": "date", "format": "strict_date_optional_time" },
      "tags":        { "type": "keyword" },
      "location":    { "type": "geo_point" },
      "embedding":   { "type": "dense_vector", "dims": 768 },
      "attributes":  { "type": "object" },
      "variants": {
        "type": "nested",
        "properties": {
          "color": { "type": "keyword" },
          "stock": { "type": "integer" }
        }
      }
    }
  }
}

Key field types

object vs nested

object fields are flattened — cross-field correlation is lost:

variants.color: [red, blue]
variants.stock: [10, 0]

ES cannot tell that red → 10 and blue → 0 are pairs. Use nested when you need to query object arrays as independent documents.

Analyzer chain

graph LR
    classDef input fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef charf fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef tok fill:#2980b9,stroke:#1f618d,color:#fff
    classDef filt fill:#27ae60,stroke:#1e8449,color:#fff

    IN["Input: 'The quick brown fox!'"]:::input --> CF["char_filter: strip punctuation<br/>→ 'The quick brown fox'"]:::charf
    CF --> TOK["tokenizer: standard<br/>→ [The, quick, brown, fox]"]:::tok
    TOK --> LC["token_filter: lowercase<br/>→ [the, quick, brown, fox]"]:::filt
    LC --> STOP["token_filter: stop<br/>→ [quick, brown, fox]"]:::filt
    STOP --> STEM["token_filter: stemmer<br/>→ [quick, brown, fox]"]:::filt
    STEM --> OUT["Terms stored in inverted index"]

Order matters here: lowercase has to run before stop, because the stop-word list is lowercase (the, not The) — swap the order and "The" never matches the filter and survives into the index as noise.

11. Analysis

Standard analyzer (default)

Tokenizes on whitespace/punctuation, lowercases, removes some punctuation. No stemming.

Custom analyzer

PUT /my_index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "html_strip": { "type": "html_strip" }
      },
      "tokenizer": {
        "my_tokenizer": { "type": "standard" }
      },
      "filter": {
        "my_stemmer": { "type": "stemmer", "language": "english" },
        "my_stop":    { "type": "stop", "stopwords": "_english_" }
      },
      "analyzer": {
        "my_analyzer": {
          "type":        "custom",
          "char_filter": ["html_strip"],
          "tokenizer":   "my_tokenizer",
          "filter":      ["lowercase", "my_stop", "my_stemmer"]
        }
      }
    }
  }
}

_analyze API — debug tokenization

GET /my_index/_analyze
{
  "analyzer": "my_analyzer",
  "text": "The <b>Quick</b> Brown Foxes!"
}

Response shows each token, its position, and offset — essential for debugging why a search isn't matching.

12. Query DSL

match — full-text search

GET /products/_search
{
  "query": {
    "match": {
      "name": { "query": "quick brown", "operator": "and" }
    }
  }
}

term — exact match (keyword fields)

{ "term": { "sku": "ABC-123" } }

range

{ "range": { "price": { "gte": 10, "lte": 100 } } }
{ "range": { "created_at": { "gte": "now-7d/d", "lt": "now/d" } } }

bool — combining queries

{
  "query": {
    "bool": {
      "must":     [{ "match": { "name": "laptop" } }],
      "filter":   [{ "term": { "in_stock": true } }, { "range": { "price": { "lte": 2000 } } }],
      "should":   [{ "term": { "brand": "apple" } }],
      "must_not": [{ "term": { "discontinued": true } }],
      "minimum_should_match": 0
    }
  }
}

filter clauses do not affect relevance score and are cached — always use filter for exact/range matches.

nested

{
  "query": {
    "nested": {
      "path": "variants",
      "query": {
        "bool": {
          "must": [
            { "term": { "variants.color": "red" } },
            { "range": { "variants.stock": { "gt": 0 } } }
          ]
        }
      }
    }
  }
}

function_score — custom relevance

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "laptop" } },
      "functions": [
        { "field_value_factor": { "field": "rating", "factor": 1.2, "modifier": "sqrt" } },
        { "gauss": { "created_at": { "origin": "now", "scale": "30d", "decay": 0.5 } } }
      ],
      "score_mode": "multiply",
      "boost_mode": "multiply"
    }
  }
}

script_score — arbitrary scoring

{
  "query": {
    "script_score": {
      "query": { "match_all": {} },
      "script": { "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
                  "params": { "query_vector": [0.1, 0.2, ...] } }
    }
  }
}

13. Aggregations

Aggregations run alongside queries. Always use filter context queries to limit the agg dataset.

terms — group by field

{
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand", "size": 10 },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}

date_histogram — time series

{
  "aggs": {
    "sales_over_time": {
      "date_histogram": { "field": "created_at", "calendar_interval": "1d" },
      "aggs": {
        "revenue": { "sum": { "field": "price" } }
      }
    }
  }
}

cardinality — unique count (HyperLogLog)

{ "aggs": { "unique_users": { "cardinality": { "field": "user_id", "precision_threshold": 1000 } } } }

Approximate — error ~0.5% at default precision. Exact cardinality requires loading all values into memory.

percentiles — latency distribution

{ "aggs": { "latency_pcts": { "percentiles": { "field": "response_ms", "percents": [50, 95, 99] } } } }

pipeline aggregations

{
  "aggs": {
    "daily_revenue": {
      "date_histogram": { "field": "date", "calendar_interval": "1d" },
      "aggs": { "revenue": { "sum": { "field": "price" } } }
    },
    "revenue_moving_avg": {
      "moving_avg": { "buckets_path": "daily_revenue>revenue", "window": 7 }
    },
    "revenue_derivative": {
      "derivative": { "buckets_path": "daily_revenue>revenue" }
    }
  }
}

14. Performance Tuning

Bulk indexing

Never index one doc at a time. Use _bulk API.

POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Laptop", "price": 999 }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Phone", "price": 699 }

Optimal bulk size: 5–15 MB per request, not by doc count. Test and tune.

Bulk indexing settings

PUT /products/_settings
{
  "index": {
    "refresh_interval": "-1",
    "number_of_replicas": "0"
  }
}

After bulk load, restore:

PUT /products/_settings
{
  "index": {
    "refresh_interval": "1s",
    "number_of_replicas": "1"
  }
}
POST /products/_forcemerge?max_num_segments=1

doc_values vs fielddata

Never enable fielddata: true on text fields in production — causes heap pressure. Instead, use a keyword sub-field for aggregations:

"name": {
  "type": "text",
  "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
}

Index sorting

Pre-sort segments for common sort patterns, speeds up queries and early termination:

PUT /logs
{
  "settings": {
    "index.sort.field": ["@timestamp"],
    "index.sort.order": ["desc"]
  }
}

Heap sizing

• Set Xms = Xmx (avoid heap resizing)
• Max 31 GB — above this, JVM can't use compressed ordinary object pointers (COOPs), memory usage jumps
• Use 50% of RAM for ES heap, leave the rest for OS page cache (Lucene uses it heavily)

# elasticsearch.yml / jvm.options
-Xms16g
-Xmx16g

Shard sizing

• Target 20–50 GB per shard
• Too many small shards: overhead per shard (metadata, threads, memory)
• Rule: ceil(total_data_GB / 30) = num_primary_shards
• For time-series data: use ILM with rollover to keep shard sizes bounded

15. Common Issues

# Profile slow queries
GET /products/_search
{
  "profile": true,
  "query": { "match": { "name": "laptop" } }
}

# Check circuit breakers
GET /_nodes/stats/breaker

# Check disk watermarks
GET /_cluster/settings

16. Index Lifecycle Management (ILM)

ILM automates moving indices through hot/warm/cold/delete phases based on age or size — the goal is to keep expensive resources (fast disk, full replica counts, CPU for scoring) allocated only to data that's actively hot, and shrink the footprint of everything older automatically instead of an operator manually re-provisioning indices every week.

graph LR
    classDef hot fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef warm fill:#f39c12,stroke:#ba6018,color:#fff
    classDef cold fill:#3498db,stroke:#2471a3,color:#fff
    classDef del fill:#7f8c8d,stroke:#616a6b,color:#fff

    subgraph POLICY["ILM policy: logs_policy"]
        HOT["Hot Phase<br/>active writes + searches<br/>priority: 100"]:::hot
        WARM["Warm Phase<br/>read-only, shrink to 1 shard<br/>forcemerge, reduced replicas<br/>priority: 50"]:::warm
        COLD["Cold Phase<br/>mounted from snapshot<br/>searchable, minimal resources<br/>priority: 0"]:::cold
        DEL["Delete Phase<br/>index removed"]:::del
    end

    HOT -->|"rollover at<br/>max_size: 50gb OR max_age: 30d"| WARM
    WARM -->|"min_age: 60d"| COLD
    COLD -->|"min_age: 180d"| DEL

ILM Policy

PUT /_ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "30d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": { "number_of_replicas": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "my_s3_repo" },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": { "delete": {} }
      }
    }
  }
}

Index template with ILM

PUT /_index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs_policy",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

Bootstrap the first index

PUT /logs-000001
{
  "aliases": {
    "logs": { "is_write_index": true }
  }
}

Write to the logs alias — ILM rolls over automatically creating logs-000002, logs-000003, etc.

Check ILM status

GET /logs-*/_ilm/explain
GET /_ilm/status

Quick Reference

# Cluster health
GET /_cluster/health?level=shards

# Node stats
GET /_nodes/stats?metric=jvm,indices,os

# Index stats
GET /products/_stats

# Pending tasks
GET /_cluster/pending_tasks

# Hot threads
GET /_nodes/hot_threads

# Flush all
POST /_flush

# Force merge (run off-peak)
POST /products/_forcemerge?max_num_segments=1

17. Relevance Scoring — BM25

ES uses BM25 (Best Match 25) by default since ES 5.0. Understanding it explains why results rank the way they do.

BM25 Formula

score(q, d) = Σ IDF(qi) * TF(qi, d)

IDF(t) = log(1 + (N - df + 0.5) / (df + 0.5))
         N  = total documents in index
         df = documents containing term t
         High IDF = rare term = more discriminating

TF(t, d) = (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * |d| / avgdl))
           freq  = term frequency in document
           k1    = term frequency saturation (default 1.2) — diminishing returns on repetition
           b     = field length normalization (default 0.75) — shorter docs rank higher
           |d|   = document field length
           avgdl = average field length across index

Key intuition:

• IDF: "laptop" in 100/1M docs scores higher than "the" in 900K/1M docs
• TF saturation: mentioning "laptop" 10x vs 5x barely matters (k1 controls this)
• b=0.75: a short product title matching "laptop" ranks higher than a long description matching "laptop"

Tuning scoring

PUT /products/_mapping
{
  "properties": {
    "name":        { "type": "text", "similarity": "BM25", "boost": 3 },
    "description": { "type": "text", "similarity": "BM25", "boost": 1 }
  }
}

Explain scoring

GET /products/_explain/1
{ "query": { "match": { "name": "laptop" } } }

18. Pagination Strategies

Four different approaches trade off simplicity, cost, and consistency guarantees differently — picking the wrong one is usually invisible until an index gets large or hot.

from/size (avoid for deep pagination)

GET /_search
{ "from": 10000, "size": 10, "query": { "match_all": {} } }

ES must fetch from + size docs from every shard, sort them on the coordinating node, then discard the first from. At from=10000, ES processes 10010 docs × N shards. Max default is 10,000 (index.max_result_window).

search_after (recommended for deep pagination)

Uses the sort values of the last result as a cursor. No skip overhead.

GET /_search
{
  "size": 20,
  "query": { "match": { "category": "electronics" } },
  "sort": [{ "price": "asc" }, { "_id": "asc" }],   // must include a tiebreaker
  "search_after": [999.99, "doc_id_xyz"]              // from last page's last hit
}

Point In Time (PIT) — consistent pagination

Results can change between pages if new docs are indexed. PIT freezes a view:

# Open PIT
POST /products/_pit?keep_alive=5m

# Use PIT with search_after
GET /_search
{
  "pit": { "id": "<pit_id>", "keep_alive": "5m" },
  "sort": [{ "price": "asc" }, { "_id": "asc" }],
  "search_after": [999.99, "doc_id_xyz"]
}

# Close PIT when done
DELETE /_pit
{ "id": "<pit_id>" }

scroll (deprecated — use search_after + PIT instead)

Old approach for bulk export. Keeps a search context open server-side. Expensive at scale. Still useful for one-time full data exports:

POST /products/_search?scroll=2m
{ "size": 1000, "query": { "match_all": {} } }

POST /_search/scroll
{ "scroll": "2m", "scroll_id": "<id>" }

19. Ingest Pipelines

Pre-process documents before they're indexed. Runs on ingest nodes.

PUT /_ingest/pipeline/access_log_pipeline
{
  "description": "Parse nginx access logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IPORHOST:client_ip} .* \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:path}\" %{NUMBER:status_code:int} %{NUMBER:bytes:long}"]
      }
    },
    { "date": { "field": "timestamp", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"] } },
    { "geoip": { "field": "client_ip" } },
    { "user_agent": { "field": "user_agent" } },
    { "remove": { "field": "message" } },
    { "set": { "field": "environment", "value": "production" } }
  ],
  "on_failure": [
    { "set": { "field": "_index", "value": "failed-{{ _index }}" } }
  ]
}

# Test pipeline without indexing
POST /_ingest/pipeline/access_log_pipeline/_simulate
{
  "docs": [{ "_source": { "message": "192.168.1.1 - - [01/Jan/2026:12:00:00 +0000] \"GET /api/health HTTP/1.1\" 200 42" } }]
}

# Use pipeline on index
POST /logs/_doc?pipeline=access_log_pipeline
{ "message": "..." }

# Set default pipeline on index
PUT /logs/_settings
{ "index.default_pipeline": "access_log_pipeline" }

Common processors: grok, date, geoip, user_agent, set, remove, rename, convert, split, join, gsub (regex replace), foreach, enrich (lookup from another index), fingerprint (dedup hash).

20. k-NN / Vector Search

Used for semantic search, recommendation, image similarity. Requires dense_vector field.

PUT /articles
{
  "mappings": {
    "properties": {
      "title":     { "type": "text" },
      "embedding": {
        "type":       "dense_vector",
        "dims":       768,
        "index":      true,
        "similarity": "cosine"     // cosine | dot_product | l2_norm
      }
    }
  }
}

Exact k-NN (brute force — small datasets)

GET /articles/_search
{
  "knn": {
    "field":         "embedding",
    "query_vector":  [0.1, 0.2, ...],   // 768 dims
    "k":             10,
    "num_candidates": 100
  }
}

Approximate nearest neighbor (ANN) — uses HNSW index

ES uses HNSW (Hierarchical Navigable Small World) graphs for ANN. Orders of magnitude faster than brute force at scale.

PUT /articles
{
  "mappings": {
    "properties": {
      "embedding": {
        "type":       "dense_vector",
        "dims":       768,
        "index":      true,
        "similarity": "cosine",
        "index_options": {
          "type":          "hnsw",
          "m":             16,      // connections per node, higher = better recall, more memory
          "ef_construction": 100    // size of candidate list during indexing
        }
      }
    }
  }
}

Hybrid search — combine BM25 + vector

GET /articles/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "title": "machine learning" } }
      ]
    }
  },
  "knn": {
    "field":         "embedding",
    "query_vector":  [...],
    "k":             10,
    "num_candidates": 100,
    "boost":         0.5
  }
}

21. Runtime Fields

Compute fields at query time without reindexing. Useful for prototyping mappings or one-off calculations.

GET /logs/_search
{
  "runtime_mappings": {
    "response_time_seconds": {
      "type": "double",
      "script": {
        "source": "emit(doc['response_ms'].value / 1000.0)"
      }
    }
  },
  "query": {
    "range": { "response_time_seconds": { "gt": 1.0 } }
  },
  "fields": ["response_time_seconds"]
}

Persistent runtime field (added to mapping, no reindex):

PUT /logs/_mapping
{
  "runtime": {
    "day_of_week": {
      "type": "keyword",
      "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" }
    }
  }
}

22. Cross-Cluster Search (CCS) & Cross-Cluster Replication (CCR)

Cross-Cluster Search — query across multiple clusters

// Configure remote cluster
PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.eu_cluster.seeds": ["eu-es-node1:9300"],
    "cluster.remote.us_cluster.seeds": ["us-es-node1:9300"]
  }
}

// Query across clusters
GET /eu_cluster:logs-*,us_cluster:logs-*,logs-*/_search
{
  "query": { "range": { "@timestamp": { "gte": "now-1h" } } }
}

Cross-Cluster Replication — replicate index to another cluster

Used for disaster recovery, geo-distribution, and keeping a read replica in another region.

PUT /follower-logs/_ccr/follow
{
  "remote_cluster":  "eu_cluster",
  "leader_index":    "logs-000001",
  "settings": {
    "number_of_replicas": 1
  }
}

Follower index is read-only. Replicates ops from leader in near-real-time. To promote follower to leader (DR failover):

POST /follower-logs/_ccr/pause_follow
POST /follower-logs/_close
POST /follower-logs/_ccr/unfollow
# follower is now a normal writable index

23. Snapshot & Restore

Four stages, from one-time setup to ongoing automation:

Register a repository (S3)

PUT /_snapshot/my_s3_repo
{
  "type": "s3",
  "settings": {
    "bucket":   "my-es-snapshots",
    "region":   "us-east-1",
    "base_path": "elasticsearch/backups"
  }
}

Take snapshot

PUT /_snapshot/my_s3_repo/snapshot_2026_01_01
{
  "indices":            "products,users",
  "include_global_state": false,
  "metadata": { "taken_by": "ops-team", "reason": "pre-migration" }
}

// Check status
GET /_snapshot/my_s3_repo/snapshot_2026_01_01

Why snapshots are cheap to repeat: Lucene segments are immutable, so a second snapshot of the same index only has to upload segments created since the last snapshot — unchanged segments are simply referenced, not re-uploaded. That's what makes daily (or hourly) snapshots practical at scale instead of a full-copy operation every time.

Restore

POST /_snapshot/my_s3_repo/snapshot_2026_01_01/_restore
{
  "indices": "products",
  "rename_pattern":     "(.+)",
  "rename_replacement": "restored_$1"     // restore as "restored_products"
}

Automated snapshots with SLM (Snapshot Lifecycle Management)

PUT /_slm/policy/daily_snapshots
{
  "schedule":   "0 0 2 * * ?",            // daily at 02:00
  "name":       "<daily-snap-{now/d}>",
  "repository": "my_s3_repo",
  "config": {
    "indices":              ["*"],
    "include_global_state": true
  },
  "retention": {
    "expire_after":   "30d",
    "min_count":      5,
    "max_count":      30
  }
}

// Execute immediately
POST /_slm/policy/daily_snapshots/_execute

24. Security

TLS + Authentication

# elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

# Generate certs
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

# Set built-in user passwords
./bin/elasticsearch-setup-passwords interactive

RBAC — Role-based access control

PUT /_security/role/logs_reader
{
  "indices": [{
    "names":      ["logs-*"],
    "privileges": ["read", "view_index_metadata"]
  }]
}

PUT /_security/user/bob
{
  "password": "changeme",
  "roles":    ["logs_reader"],
  "full_name": "Bob Smith"
}

Field-level and document-level security

PUT /_security/role/restricted_reader
{
  "indices": [{
    "names":      ["orders"],
    "privileges": ["read"],
    "field_security": {
      "grant": ["order_id", "status", "created_at"]   // only these fields visible
    },
    "query": "{ \"term\": { \"region\": \"EU\" } }"   // only EU docs visible
  }]
}

25. Transforms & Rollups

Transforms — materialize aggregations into a new index

PUT /_transform/daily_sales_summary
{
  "source": { "index": "orders" },
  "dest":   { "index": "orders_daily" },
  "pivot": {
    "group_by": {
      "date":     { "date_histogram": { "field": "created_at", "calendar_interval": "1d" } },
      "category": { "terms": { "field": "category" } }
    },
    "aggregations": {
      "total_revenue": { "sum": { "field": "price" } },
      "order_count":   { "value_count": { "field": "_id" } }
    }
  },
  "sync": {
    "time": { "field": "created_at", "delay": "60s" }   // continuous transform
  }
}

POST /_transform/daily_sales_summary/_start

Transforms replace rollups (deprecated). Use them for pre-aggregated dashboards, summary indexes, and reducing query load on high-cardinality indexes.