AlertManager

Prometheus decides what is wrong; AlertManager decides who hears about it, how often, and whether it's worth waking someone up. It sits between rule evaluation and a human's phone — deduplicating identical alerts, grouping related ones into a single notification, routing by label match, and suppressing noise via inhibition and silences.

1. Architecture

Prometheus evaluates rules and pushes firing alerts to AlertManager, which deduplicates, groups, routes, and dispatches notifications.

flowchart LR
    P[Prometheus] -->|POST /alerts| AM[AlertManager]
    AM --> RT{Routing Tree}
    RT -->|severity=critical| PD[PagerDuty]
    RT -->|severity=warning| SL[Slack]
    RT -->|default| EM[Email]

2. Alert Lifecycle

An alert transitions through states based on the for: duration in the rule and whether it resolves.

stateDiagram-v2
    [*] --> Inactive: rule not firing
    Inactive --> Pending: condition true
    Pending --> Firing: for-duration met
    Pending --> Inactive: condition false
    Firing --> Resolved: condition false
    Resolved --> [*]

3. Routing Tree

Routes are evaluated top-down; first match wins. Each route can override receiver, group_by, and timing.

flowchart TD
    G[global defaults] --> R[root route]
    R --> M1{team=infra?}
    M1 -->|yes| R1[pagerduty-infra]
    M1 -->|no| M2{severity=warning?}
    M2 -->|yes| R2[slack-warning]
    M2 -->|no| R3[default receiver]

Config example:

route:
  receiver: default-receiver
  group_by: [alertname, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = critical
      receiver: pagerduty-critical
      continue: false
    - matchers:
        - severity = warning
      receiver: slack-warning

4. Grouping

Grouping prevents alert storms: 100 pods failing → 1 grouped notification.

sequenceDiagram
    participant P as Prometheus
    participant AM as AlertManager
    participant R as Receiver

    P->>AM: Alert A fires (group: cluster=prod)
    Note over AM: group_wait timer starts (30s)
    P->>AM: Alert B fires, same group
    Note over AM: group_wait elapses, batch A+B
    AM->>R: notify (A, B)
    P->>AM: Alert C fires, same group
    Note over AM: waits group_interval (5m) before adding new alerts
    AM->>R: notify (A, B, C)
    Note over AM: A, B, C still firing after repeat_interval (4h)
    AM->>R: re-notify (A, B, C)

5. Inhibition

Suppress derived/symptom alerts when the root cause is already firing. Inhibition rules match a source alert and silence matching target alerts.

inhibit_rules:
  - source_matchers:
      - alertname = "NodeDown"
    target_matchers:
      - alertname =~ "Pod.*"
    equal: [cluster, node]

If NodeDown fires for node=worker-1, all Pod* alerts on the same node are silenced.

6. Silences

Silences mute alerts matching a set of matchers for a time window. Created via UI or amtool.

Types:

# Create silence for 2 hours on a specific service
amtool silence add alertname="HighErrorRate" service="payments" \
  --duration=2h --comment="Deploying payments v2.3"

# List active silences
amtool silence query

# Expire a silence
amtool silence expire <silence-id>

Grouping, inhibition, and silencing all reduce notification noise, but they solve different problems — worth keeping straight:

7. Complete alertmanager.yml

This ties every piece above into one file — global defaults, the routing tree, receivers, and inhibition rules together. The receivers block wires up three different notification channels to the same AlertManager instance:

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'secret'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

templates:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  receiver: default-email
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h
      continue: false

    - matchers:
        - severity = warning
      receiver: slack-warning
      group_wait: 1m
      repeat_interval: 8h
      continue: false

    - matchers:
        - alertname = Watchdog
      receiver: null-receiver

receivers:
  - name: null-receiver

  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: '<PAGERDUTY_INTEGRATION_KEY>'
        description: '{{ template "pagerduty.default.description" . }}'
        severity: critical
        details:
          firing: '{{ .Alerts.Firing | len }}'
          cluster: '{{ .CommonLabels.cluster }}'

  - name: slack-warning
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts-warning'
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: >-
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Severity:* {{ .Labels.severity }}
          *Details:* {{ range .Labels.SortedPairs }} {{ .Name }}={{ .Value }} {{ end }}
          {{ end }}
        send_resolved: true

  - name: default-email
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true

inhibit_rules:
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: [alertname, cluster, namespace]

  - source_matchers:
      - alertname = NodeDown
    target_matchers:
      - alertname =~ "Pod.*"
    equal: [node]

8. Debugging

Config validation:

amtool check-config /etc/alertmanager/alertmanager.yml

Query active alerts:

# All firing alerts
amtool alert query

# Filter by label
amtool alert query severity=critical

# Against a specific AlertManager
amtool alert query --alertmanager.url=http://alertmanager:9093

API endpoints:

# List all alerts (v2 API)
curl http://alertmanager:9093/api/v2/alerts | jq .

# List active silences
curl http://alertmanager:9093/api/v2/silences | jq .

# AlertManager status
curl http://alertmanager:9093/api/v2/status | jq .

# Reload config (SIGHUP or POST)
curl -X POST http://alertmanager:9093/-/reload

Common issues: