Thanos and Mimir — Long-Term Metrics Storage

Prometheus stores data locally for 15 days by default. For long-term retention, global query across clusters, and high availability, you need Thanos or Mimir.

The Problem Prometheus Alone Can't Solve

Two different answers to that problem — flip between them before going deeper into either:

Thanos

Thanos extends Prometheus by adding components that bolt onto existing Prometheus instances.

graph TD
    subgraph Cluster A
        PA[Prometheus A] -->|sidecar reads WAL| SA[Thanos Sidecar A]
        SA -->|upload blocks| S3[(S3 / GCS)]
    end
    subgraph Cluster B
        PB[Prometheus B] -->|sidecar reads WAL| SB[Thanos Sidecar B]
        SB -->|upload blocks| S3
    end

    QF[Thanos Query Frontend] --> Q[Thanos Querier]
    Q -->|fan-out query| SA
    Q -->|fan-out query| SB
    Q -->|query historical| ST[Thanos Store Gateway]
    ST -->|reads blocks| S3

    COMP[Thanos Compactor] -->|compact + downsample| S3
    RULE[Thanos Ruler] -->|global recording/alert rules| Q

Components

A single query touches every one of these pieces at once — here's one PromQL query's path through the fan-out:

sequenceDiagram
    participant G as Grafana
    participant QF as Query Frontend
    participant Q as Querier
    participant S as Sidecars
    participant SG as Store Gateway
    G->>QF: PromQL query
    QF->>QF: split into sub-queries, check cache
    QF->>Q: forward sub-queries
    Q->>S: fan-out, live data
    Q->>SG: fan-out, historical data from object storage
    S-->>Q: recent series
    SG-->>Q: historical series
    Q->>Q: merge, dedupe by replica label
    Q-->>QF: merged result
    QF-->>G: response

Deduplication

When two Prometheus replicas scrape the same targets (HA setup), both upload identical data. Thanos Querier deduplicates by matching replica label:

# Prometheus HA: two replicas, identical config except replica label
global:
  external_labels:
    cluster: production
    replica: "0"   # replica "1" on the other instance

# Query with deduplication
thanos query --query.replica-label=replica
# Querier merges series with identical labels except the replica label

Object Storage Config

# thanos-storage-config.yaml (mounted as secret)
type: S3
config:
  bucket: my-thanos-metrics
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
  # Uses IRSA on EKS — no access key needed

Downsampling

Thanos Compactor creates downsampled versions of old data:

# Grafana/Thanos auto-selects resolution based on query range
# You can also explicitly request:
# step=5m → Thanos picks 5m downsampled if range > 40h

Here's what happens to a single data point as it ages past each threshold:

Mimir (Grafana's Prometheus-compatible, horizontally scalable backend)

Mimir is a fully distributed, horizontally scalable TSDB. It implements the Prometheus remote_write and query API — drop-in replacement at scale.

graph LR
    subgraph Ingest Path
        RW[Prometheus remote_write] --> DIST[Distributor]
        DIST -->|hash ring| ING[Ingester x3]
        ING -->|flush| S3[(S3 object store)]
    end
    subgraph Query Path
        GF[Grafana] -->|PromQL| QF2[Query Frontend]
        QF2 -->|split + cache| QUER[Querier]
        QUER -->|recent data| ING
        QUER -->|historical| SG[Store Gateway]
        SG --> S3
    end
    subgraph Background
        COMP2[Compactor] --> S3
        RULER2[Ruler] --> QUER
    end

Key differences vs Thanos

Mimir Multi-Tenancy

# Prometheus remote_write with tenant header
remote_write:
  - url: http://mimir-distributor:8080/api/v1/push
    headers:
      X-Scope-OrgID: "team-platform"

# Query with tenant
curl -H "X-Scope-OrgID: team-platform" \
  "http://mimir-query-frontend:8080/prometheus/api/v1/query?query=up"

Choosing Between Them

Small-medium (1-5 clusters, < 5M series, < 1 year retention)
  → Thanos with Sidecar + S3 + Store Gateway
  → Simplest to operate, reuses existing Prometheus

Large (5+ clusters, 5-50M series, multi-tenant)
  → Mimir
  → Purpose-built for scale

SaaS / platform team serving many teams
  → Mimir (multi-tenancy built-in)

Already on Grafana Cloud
  → Grafana Mimir (hosted)

The same decision, as a flowchart:

graph TD
    Q{"What's the scale?"}
    Q -->|"1-5 clusters, under 5M series, under 1yr retention"| T["Thanos: Sidecar + S3 + Store Gateway<br/>simplest to operate, reuses existing Prometheus"]
    Q -->|"5+ clusters, 5-50M series, multi-tenant"| M1["Mimir<br/>purpose-built for scale"]
    Q -->|"SaaS / platform team serving many teams"| M2["Mimir<br/>multi-tenancy built-in"]
    Q -->|"already on Grafana Cloud"| M3["Grafana Mimir (hosted)"]

Hands-on: Thanos on EKS

# 1. Add external_labels to existing Prometheus ConfigMap
kubectl edit configmap prometheus-config -n monitoring
# Add:
# global:
#   external_labels:
#     cluster: production-us-east-1
#     replica: "0"

# 2. Deploy Thanos Sidecar alongside Prometheus
# (add as second container in Prometheus Deployment)
# - shares /data volume with Prometheus
# - uploads blocks every 2h

# 3. Create S3 bucket for blocks
aws s3 mb s3://my-thanos-blocks --region us-east-1

# 4. Deploy Store Gateway (reads historical from S3)
# 5. Deploy Querier (fan-out to Sidecar + Store Gateway)
# 6. Point Grafana at Thanos Querier instead of Prometheus directly

# Verify blocks are uploading
kubectl logs -n monitoring deployment/thanos-sidecar | grep "uploaded"

# Check Thanos Querier sees all stores
thanos tools bucket inspect \
  --objstore.config-file=storage.yaml \
  --output=table

# Query via Thanos (same PromQL as Prometheus)
curl "http://thanos-querier:9090/api/v1/query?query=up"

Retention and Cost

Raw Prometheus (15 days, local disk):  $0.10/GB/month (EBS gp3)
Thanos on S3 (1 year):                $0.023/GB/month (S3 Standard)
  + downsampling reduces query cost for old data
  + lifecycle rules: move to S3-IA after 30 days → $0.0125/GB/month

Typical savings: 80-90% vs keeping data on Prometheus local disk

# S3 lifecycle rule for cost optimization
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-thanos-blocks \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "move-to-ia",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"}]
    }]
  }'