❯devopsindex

OpenTelemetry

OpenTelemetry (OTEL) is the vendor-neutral standard for generating telemetry — one set of SDKs and one wire protocol (OTLP) work across languages and backends, so instrumenting a service once doesn't lock it into a single traces/metrics/logs vendor. This guide covers the three signal types, how a trace's context propagates across a service boundary, the Collector's receive → process → export pipeline, and where to draw the line between auto- and manual instrumentation.

0/0 checks

Three Pillars

Signal What it is Backend
Traces End-to-end request flow across services Jaeger, Tempo, Zipkin
Metrics Numeric measurements over time Prometheus, Mimir
Logs Structured event records Loki, Elasticsearch

Correlation — inject trace_id and span_id into structured logs so a log line links back to the exact trace:

{"level":"error","trace_id":"4bf92f3577b34da6","span_id":"00f067aa0ba902b7","msg":"db timeout"}

Why do you inject trace_id and span_id into structured logs?

Architecture

graph LR
    APP["Go App<br/>OTEL SDK"] -->|OTLP gRPC/HTTP| COL[OTEL Collector]
    COL -->|traces| JAE["Jaeger / Tempo"]
    COL -->|metrics| PROM[Prometheus]
    COL -->|logs| LOKI[Loki]
    JAE --> GR[Grafana]
    PROM --> GR
    LOKI --> GR

Trace Propagation

sequenceDiagram
    participant C as Client
    participant A as Service A
    participant B as Service B
    participant DB as Database

    C->>A: HTTP GET /order/42<br/>traceparent: 00-traceId-spanA-01
    A->>B: gRPC GetProduct<br/>traceparent: 00-traceId-spanB-01
    B->>DB: SQL SELECT
    DB-->>B: rows
    B-->>A: product data
    A-->>C: 200 OK
    Note over C,DB: All spans share same traceId

Same call, one step at a time:

1. Client → Service A. The client issues HTTP GET /order/42. Service A's span for this request carries traceparent: 00-traceId-spanA-01 — a traceId for the whole request, plus this hop's own span, spanA.
2. Service A → Service B. Service A calls Service B over gRPC (GetProduct), forwarding traceparent: 00-traceId-spanB-01 — same traceId carried across the boundary, a new span (spanB) for this hop.
3. Service B → Database. Service B issues a SQL SELECT; the database returns rows back to B.
4. Unwind. B returns product data to A, A returns 200 OK to the client. Per the diagram's note, every span in this call — client, A, B, DB — shares the same traceId, which is what lets a backend group them into a single trace.

Span Anatomy

TraceID: 4bf92f3577b34da6a3ce929d0e0e4736
SpanID:  00f067aa0ba902b7
Parent:  a3ce929d0e0e4736   (nil for root span)
Name:    "HTTP GET /order/{id}"
Start:   2024-01-01T10:00:00.000Z
End:     2024-01-01T10:00:00.043Z
Status:  OK
Attributes:
  http.method = GET
  http.url    = /order/42
  http.status_code = 200
Events:
  - name: "cache miss", timestamp: T+2ms
Baggage: user.tier=premium  (propagated to all downstream spans)

What's the difference between an Attribute and Baggage on a span, per the anatomy above?

Sampling Strategies

Strategy How Use
Head (probabilistic) Decision at root span; 1-10% sampled Low overhead, misses rare errors
Head (rate-limiting) Max N traces/sec Predictable cost
Tail Buffer all spans; decide after root completes Can sample 100% of errors
Parent-based Inherit parent's sampling decision Consistent across services
Decision made at the root span, before the outcome is known — typically 1-10% sampled. Low overhead, but it misses rare errors since a failing request has the same small chance of being sampled as any other.
Cap of max N traces/sec, decided at the root span. Predictable cost regardless of traffic spikes.
Buffers all spans and decides only after the root span completes — so it can specifically keep every error trace. Can sample 100% of errors, at the cost of buffering overhead.
Each span inherits whatever sampling decision its parent already made. Keeps sampling consistent across services — a span never gets sampled on its own while its parent was dropped, or vice versa.

Which sampling strategy can guarantee capturing 100% of error traces, and why can't head-based sampling promise that?

Metrics: OTEL vs Prometheus

Aspect OTEL Prometheus
Data model OTLP (protobuf) Text exposition
Push vs pull Push (to collector) Pull (scrape)
Histogram Explicit bounds or exponential Fixed buckets
Exemplars Built-in (link metrics → trace) Supported
Aggregation In SDK or collector In PromQL
Temporality Delta or cumulative Cumulative only

OTEL metrics exported via Prometheus exporter endpoint look identical to native Prometheus metrics.

OTEL metrics support both delta and cumulative temporality. What does Prometheus support, and does that difference disappear once OTEL metrics reach a Prometheus exporter endpoint?

Auto vs Manual Instrumentation

Auto-instrumentation — zero-code, wraps stdlib/frameworks:

// HTTP auto-instrumentation via otelhttp
mux := http.NewServeMux()
handler := otelhttp.NewHandler(mux, "my-service")
http.ListenAndServe(":8080", handler)

// Database via otelsql
db, _ = otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL))

Manual instrumentation — full control over span names, attributes, events:

ctx, span := tracer.Start(ctx, "processOrder",
    trace.WithAttributes(
        attribute.String("order.id", orderID),
        attribute.Int("order.items", len(items)),
    ),
)
defer span.End()

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}
span.AddEvent("payment.processed", trace.WithAttributes(
    attribute.String("payment.method", "card"),
))
Zero-code, wraps stdlib/frameworks — otelhttp.NewHandler for HTTP, otelsql.Open for database calls. Fast to add, but you only get whatever spans and attributes the wrapper already knows to record.
Full control over span names, attributes, and events — call tracer.Start directly, attach custom attributes like order.id, record errors with span.RecordError/SetStatus, and add business-specific events like payment.processed.

What can manual instrumentation give you that auto-instrumentation via otelhttp/otelsql doesn't?

Go OTEL SDK Example

package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel/exporters/prometheus"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exp, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("order-service"),
        semconv.ServiceVersion("1.0.0"),
    )
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1))), // 10% sampling
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

func initMeter() (*sdkmetric.MeterProvider, error) {
    exp, err := prometheus.New()
    if err != nil {
        return nil, err
    }
    mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(exp))
    otel.SetMeterProvider(mp)
    return mp, nil
}

func main() {
    ctx := context.Background()

    tp, _ := initTracer(ctx)
    defer tp.Shutdown(ctx)

    mp, _ := initMeter()
    defer mp.Shutdown(ctx)

    tracer := otel.Tracer("order-service")
    meter  := otel.Meter("order-service")

    reqCounter, _ := meter.Int64Counter("http.requests.total",
        metric.WithDescription("Total HTTP requests"))

    latency, _ := meter.Float64Histogram("http.request.duration",
        metric.WithUnit("s"))

    mux := http.NewServeMux()
    mux.HandleFunc("/order", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        ctx, span := tracer.Start(r.Context(), "handle.order",
            trace.WithAttributes(attribute.String("http.method", r.Method)),
        )
        defer span.End()

        // business logic ...
        processOrder(ctx, tracer)

        reqCounter.Add(ctx, 1, metric.WithAttributes(
            attribute.String("method", r.Method),
            attribute.Int("status", 200),
        ))
        latency.Record(ctx, time.Since(start).Seconds())
        w.WriteHeader(http.StatusOK)
    })

    // Wrap with OTEL HTTP middleware (adds server span automatically)
    http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "order-service"))
}

func processOrder(ctx context.Context, tracer trace.Tracer) {
    _, span := tracer.Start(ctx, "processOrder")
    defer span.End()
    time.Sleep(10 * time.Millisecond)
}

OTEL Collector Config

graph LR
    APP[App OTLP] --> RCV[Receivers]
    PROM_SC["Prometheus<br/>scrape"] --> RCV
    RCV --> PROC["Processors<br/>batch/filter/attr"]
    PROC --> EXP[Exporters]
    EXP --> JAE[Jaeger]
    EXP --> PR[Prometheus]
    EXP --> LK[Loki]

The pipeline is three stages, one step at a time:

1. Receive. Data arrives two ways: OTLP push from instrumented apps, or a Prometheus-style scrape — both land in Receivers.
2. Process. Receivers hand off to Processors: memory_limiter caps the collector's own memory (limit_mib), filter/drop_debug drops noisy spans (e.g. the /healthz health-check target), resource adds attributes like env=production, and batch groups records before export.
3. Export. Processed data reaches Exporters, which fan out per signal type — traces to Jaeger, metrics to Prometheus, logs to Loki. 
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: "self"
          static_configs:
            - targets: ["localhost:8888"]

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
  resource:
    attributes:
      - key: env
        value: production
        action: insert
  filter/drop_debug:
    traces:
      span:
        - 'attributes["http.target"] == "/healthz"'

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, filter/drop_debug, batch]
      exporters:  [jaeger, otlp/tempo]
    metrics:
      receivers:  [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters:  [prometheus]
    logs:
      receivers:  [otlp]
      processors: [batch]
      exporters:  [loki]

Which processor in the collector config drops health-check spans, and which one caps the collector's own memory usage?

← Previous
Grafana
Next →
Loki Log Aggregation