Service Mesh (Istio / Linkerd)

Sidecar-proxied traffic management, mTLS, and observability for a fleet of microservices — without touching application code. Track how many knowledge checks you clear as you go:

1. The Problem

With N microservices, cross-cutting concerns appear in every service:

2. Architecture

graph TD
    subgraph CP["Control Plane"]
        istiod["istiod<br/>(Pilot+Citadel+Galley)"]
    end

    subgraph DP["Data Plane"]
        subgraph PodA["Pod A"]
            appA["App Container"]
            envoyA["Envoy Sidecar"]
        end
        subgraph PodB["Pod B"]
            appB["App Container"]
            envoyB["Envoy Sidecar"]
        end
    end

    istiod -->|xDS config| envoyA
    istiod -->|xDS config| envoyB
    istiod -->|issue certs| envoyA
    istiod -->|issue certs| envoyB
    envoyA -->|mTLS traffic| envoyB
    appA -->|localhost| envoyA
    envoyB -->|localhost| appB

• istiod combines Pilot (service discovery, xDS), Citadel (cert authority), Galley (config validation)
• Envoy sidecars intercept all inbound/outbound traffic via iptables rules injected by the init container
• xDS APIs (LDS, RDS, CDS, EDS) push config to proxies without restart

3. Traffic Management

VirtualService — routing rules

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
  - match:
    - headers:
        end-user:
          exact: test-user
    route:
    - destination:
        host: reviews
        subset: v2
  - route:                      # default: canary split
    - destination:
        host: reviews
        subset: v1
      weight: 90
    - destination:
        host: reviews
        subset: v2
      weight: 10

DestinationRule — subset definitions

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Traffic Flow with Sidecar

sequenceDiagram
    participant C as Client Pod<br/>(Envoy)
    participant VS as VirtualService<br/>Rule
    participant S1 as reviews-v1<br/>(Envoy)
    participant S2 as reviews-v2<br/>(Envoy)

    C->>VS: HTTP GET /reviews
    VS-->>C: route: 90% v1 / 10% v2
    C->>S1: mTLS (90% traffic)
    C->>S2: mTLS (10% traffic)
    S1-->>C: response
    S2-->>C: response

4. Security

mTLS — PeerAuthentication

# STRICT: only mTLS accepted
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT   # or PERMISSIVE (plain+mTLS)

AuthorizationPolicy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-reviews
  namespace: production
spec:
  selector:
    matchLabels:
      app: reviews
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/productpage"]
    to:
    - operation:
        methods: ["GET"]

mTLS Handshake

sequenceDiagram
    participant EA as Envoy A<br/>(client sidecar)
    participant CA as istiod CA
    participant EB as Envoy B<br/>(server sidecar)

    EA->>CA: CSR (SPIFFE SVID)
    CA-->>EA: signed cert
    EB->>CA: CSR (SPIFFE SVID)
    CA-->>EB: signed cert
    EA->>EB: TLS ClientHello
    EB-->>EA: TLS ServerHello + cert
    EA->>EB: verify cert (SPIFFE ID)
    EB-->>EA: mutual verify done
    EA->>EB: encrypted app traffic

SPIFFE ID format: spiffe://cluster.local/ns/<namespace>/sa/<serviceaccount>

Step through what that sequence diagram is actually doing, one exchange at a time:

5. Observability

Envoy emits metrics automatically — no app instrumentation needed.

Key metrics:

istio_requests_total{source_app, destination_app, response_code}
istio_request_duration_milliseconds_bucket
istio_tcp_connections_opened_total

Access logs, distributed traces (Jaeger/Zipkin via B3 headers), and Kiali topology graph come out of the box.

6. Circuit Breaking & Retries

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: payment
spec:
  host: payment
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
    outlierDetection:             # circuit breaker
      consecutiveGatewayErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: "5xx,connect-failure"

7. Linkerd vs Istio

Rule of thumb: start with Linkerd if you just need mTLS + basic observability. Use Istio when you need fine-grained traffic policies, canary deployments, or complex AuthorizationPolicies.

8. Fault Injection — Chaos Testing via Istio

Istio can inject faults at the network level without touching application code — perfect for chaos engineering.

# Inject HTTP 503 errors for 20% of requests to reviews
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews-fault
spec:
  hosts: [reviews]
  http:
  - fault:
      abort:
        percentage:
          value: 20.0
        httpStatus: 503
    route:
    - destination:
        host: reviews
        subset: v1
---
# Inject 5-second delay for 10% of requests (test timeout handling)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings-delay
spec:
  hosts: [ratings]
  http:
  - fault:
      delay:
        percentage:
          value: 10.0
        fixedDelay: 5s
    route:
    - destination:
        host: ratings
        subset: v1

9. Timeouts and Retries

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
    timeout: 3s                     # total request timeout
    retries:
      attempts: 3
      perTryTimeout: 1s             # each attempt gets 1s (total: 3s across 3 attempts)
      retryOn: "5xx,connect-failure,reset"   # retry on these conditions

retryOn conditions:

Important: Only retry idempotent operations (GET, PUT). Never auto-retry POST without idempotency keys.

10. Traffic Shifting — Canary Deployment

sequenceDiagram
    participant USER as Users (100%)
    participant ISTIO as Istio VirtualService
    participant V1 as reviews-v1 (90%)
    participant V2 as reviews-v2 (10%)

    USER->>ISTIO: GET /reviews
    ISTIO->>V1: 90% of requests
    ISTIO->>V2: 10% of requests (canary)
    Note over ISTIO: Monitor error rate on v2
    Note over ISTIO: If error rate OK: shift to 25%, 50%, 100%
    Note over ISTIO: If error rate high: shift back to 0%

# Gradually shift traffic using kubectl patch
kubectl patch virtualservice reviews --type=json \
  -p='[{"op":"replace","path":"/spec/http/0/route/0/weight","value":75},
       {"op":"replace","path":"/spec/http/0/route/1/weight","value":25}]'

# Monitor canary via Prometheus
# istio_requests_total{destination_service="reviews",destination_version="v2",response_code!~"5.."}
# / istio_requests_total{destination_service="reviews",destination_version="v2"}
# Alert if error rate > 1% on v2 → shift back to 0%