Grafana

1. Architecture & Data Sources

Grafana is a visualization layer that queries data sources and renders panels. It does not store metrics — it proxies queries to backends.

flowchart LR
    subgraph Sources
        PR["Prometheus<br/>metrics"]
        LK["Loki<br/>logs"]
        TP["Tempo<br/>traces"]
        PG["PostgreSQL<br/>relational"]
    end
    subgraph Grafana
        DS["Data Source<br/>plugins"] --> QE[Query Engine]
        QE --> PN[Panel Renderer]
    end
    Sources --> DS
    PN --> DB[Dashboard]

A panel doesn't talk to a backend directly — every query takes the same path from the panel editor to pixels on screen:

2. Panel Types

Picking the right panel:

• Trending over time → Time series
• "Is it OK right now?" → Stat or Gauge
• "Which pods are slowest?" → Table
• "Where are latency outliers?" → Heatmap
• "What did the app log?" → Logs

3. Variables

Variables make dashboards reusable across environments, clusters, and services.

Dashboard URL: /d/abc?var-cluster=prod&var-namespace=payments

Query variable — populated from a data source:

# Variable: cluster
# Query (Prometheus label_values):
label_values(kube_node_info, cluster)

Custom variable — static list:

name: env
values: dev,staging,prod

Interval variable — for $__interval in rate() calls:

name: interval
values: 1m,5m,10m,30m
auto: true

Use in panels:

rate(http_requests_total{cluster="$cluster", namespace="$namespace"}[$interval])

4. USE Method Dashboard

Per resource (CPU, memory, disk, network):

Layout (4 rows × 3 panels):

[CPU Util]  [CPU Saturation]  [CPU Errors]
[Mem Util]  [Mem Saturation]  [OOM Kills ]
[Disk Util] [Disk Saturation] [Disk Errors]
[Net Util]  [Net Saturation]  [Net Errors ]

Each panel uses $node variable to filter by host.

5. RED Method Dashboard

Per service/endpoint:

Layout:

[RPS - all services (time series)]
[Error Rate % (time series)]        [Top errors by service (table)]
[P50 latency] [P95 latency] [P99 latency]

6. SLO Dashboard

SLO: 99.9% availability over 30-day rolling window
Error budget: 0.1% = ~43 minutes/month

Burn rate thresholds (Google SRE):

# 1-hour burn rate
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
  /
  sum(rate(http_requests_total[1h]))
) / (1 - 0.999)

7. Provisioning Dashboards as Code

flowchart LR
    GH["Git repo<br/>dashboard JSON"] -->|CI push| CM[K8s ConfigMap]
    CM -->|volume mount| SC["Grafana sidecar<br/>container"]
    SC -->|watches /dashboards| GF["Grafana<br/>auto-reloads"]

ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  labels:
    grafana_dashboard: "1"   # sidecar watches this label
data:
  red-dashboard.json: |
    { "title": "RED Dashboard", ... }

Grafana Helm values:

grafana:
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      searchNamespace: ALL
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: Prometheus
          type: prometheus
          url: http://prometheus-server:9090
          isDefault: true
        - name: Loki
          type: loki
          url: http://loki:3100

Grafana sidecar watches all ConfigMaps with grafana_dashboard: "1" and hot-reloads dashboards without restart.

8. Alerting

Grafana-native alerting (Grafana 9+) replaces the old panel-level alerts.

Components:

Contact point (Slack):

# provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: slack-oncall
    receivers:
      - uid: slack-oncall-uid
        type: slack
        settings:
          url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
          recipient: '#alerts'
          title: '{{ .CommonLabels.alertname }}'

Notification policy:

# provisioning/alerting/notification-policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: slack-oncall
    group_by: [alertname, cluster]
    routes:
      - receiver: pagerduty-critical
        matchers:
          - severity = critical

Alert rule (provisioning):

apiVersion: 1
groups:
  - orgId: 1
    name: RED Alerts
    folder: SRE
    interval: 1m
    rules:
      - title: High Error Rate
        condition: C
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error rate above 1% for {{ $labels.service }}"
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              conditions:
                - evaluator: { params: [0.01], type: gt }
                  query: { params: [A] }

A rule evaluating true once doesn't fire immediately — the for: duration exists precisely to filter out single noisy samples:

flowchart LR
    EV["Rule evaluates<br/>every interval"] -->|condition breached| PD["Pending<br/>for-duration timer starts"]
    PD -->|condition clears early| OK[Normal]
    PD -->|still breached after<br/>full for-duration| FR[Firing]
    FR --> NP["Notification policy<br/>matches alert labels"]
    NP --> CP["Contact point<br/>Slack, PagerDuty, email"]
    FR -->|condition clears| OK