TLS, SSL, and Encryption

Encryption — The Big Picture (Generic)

Before TLS specifics, understand the three types of encryption and why HTTPS needs all three.

1. Symmetric Encryption — One Shared Key

Both sides use the same key to lock and unlock.

graph LR
    classDef plain fill:#27ae60,stroke:#1e8449,color:#fff
    classDef key fill:#f39c12,stroke:#ba6018,color:#fff
    classDef cipher fill:#7f8c8d,stroke:#616a6b,color:#fff

    PLAIN1["Hello World<br/>(plaintext)"]:::plain
    KEY1["🔑 Key: ABC123<br/>same key locks and unlocks"]:::key
    ENC1["Xk39dP#!<br/>(ciphertext)"]:::cipher
    DEC1["Hello World<br/>(plaintext, recovered)"]:::plain

    PLAIN1 -->|"encrypt with KEY1"| ENC1
    ENC1 -->|"decrypt with KEY1"| DEC1
    KEY1 -.->|"same key used both ways"| ENC1
    KEY1 -.->|"same key used both ways"| DEC1

Problem: How do you share the key? If an attacker intercepts it, they can decrypt everything.

Fast: AES-256-GCM encrypts ~1 GB/s. Used for all bulk data.

2. Asymmetric Encryption — Public/Private Key Pair

Two mathematically linked keys. Public key encrypts, private key decrypts. The private key never leaves the server.

graph TD
    classDef keypair fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef good fill:#27ae60,stroke:#1e8449,color:#fff
    classDef cipher fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef bad fill:#e74c3c,stroke:#c0392b,color:#fff

    SERVER["Server generates key pair once:<br/>🔓 Public key (share with everyone)<br/>🔒 Private key (NEVER share, stays on server)"]:::keypair

    subgraph OK["Anyone can send a secret message"]
        ALICE["Alice has a secret:<br/>'my password'"]:::good
        ALICE -->|"encrypt with server's PUBLIC key 🔓"| CIPHER["Xk39dP#!<br/>(ciphertext)"]:::cipher
        CIPHER -->|"send over internet"| SERVER2["Server decrypts<br/>with PRIVATE key 🔒"]:::good
        SERVER2 --> PLAIN2["'my password'<br/>(recovered)"]:::good
    end

    subgraph BAD["Attacker intercepts but cannot decrypt"]
        ATTACKER["Eve intercepts: Xk39dP#!"]:::cipher
        ATTACKER -->|"has public key 🔓<br/>but NOT private key 🔒"| FAIL["❌ Cannot decrypt"]:::bad
    end

    SERVER -.->|"public key distributed to everyone"| ALICE
    SERVER -.->|"public key distributed to everyone"| ATTACKER

Problem: 100× slower than symmetric. Can't use for bulk data.

3. How HTTPS Combines Both (Hybrid Encryption)

HTTPS uses asymmetric only to agree on a shared key, then uses symmetric for all data:

sequenceDiagram
    participant C as Your Browser
    participant S as Server (e.g. google.com)

    rect rgb(40, 55, 75)
    Note over C,S: Step 1 — Key Exchange (Asymmetric, happens once)
    S->>C: "Here's my PUBLIC KEY 🔓 (inside the certificate)"
    C->>C: Generate a random session key: 🔑 "7f3a9c..."
    C->>S: Encrypt session key with server's PUBLIC KEY 🔓<br/>→ send "Xk8dP#!" (only server can decrypt)
    S->>S: Decrypt with PRIVATE KEY 🔒<br/>→ recovers session key 🔑 "7f3a9c..."
    end

    Note over C,S: Both now have the same session key 🔑 — nobody else does

    rect rgb(40, 60, 45)
    Note over C,S: Step 2 — Actual Data (Symmetric, blazing fast)
    C->>S: "GET /profile" encrypted with 🔑 session key
    S->>C: "200 OK {name: Alice}" encrypted with 🔑 session key
    end

    rect rgb(75, 40, 40)
    Note over C,S: If attacker intercepts step 1 — only sees encrypted session key<br/>Cannot decrypt without private key 🔒
    end

Summary of what each does in HTTPS:

Symmetric vs Asymmetric — Side by Side

graph TD
    classDef sym fill:#3498db,stroke:#2471a3,color:#fff
    classDef asym fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef hybrid fill:#27ae60,stroke:#1e8449,color:#fff

    subgraph SYM["Symmetric Encryption"]
        SA["🔑 One key<br/>Same key encrypts AND decrypts<br/>AES-256-GCM, ChaCha20<br/>Speed: ~1 GB/s<br/>Problem: how to share the key?"]:::sym
    end

    subgraph ASYM["Asymmetric Encryption"]
        AS["🔓🔒 Key pair<br/>PUBLIC key encrypts<br/>PRIVATE key decrypts<br/>RSA-2048, ECDSA, ECDH<br/>Speed: ~10 MB/s<br/>No key-sharing problem"]:::asym
    end

    subgraph HYBRID["Hybrid — what TLS actually does"]
        H1["ECDHE (asymmetric)<br/>Securely agree on a shared secret"]:::hybrid
        H2["AES-256-GCM (symmetric)<br/>Encrypt all actual data with that secret"]:::hybrid
        H1 --> H2
    end

    SYM -->|"solves speed"| HYBRID
    ASYM -->|"solves key exchange"| HYBRID

Why hybrid? Asymmetric crypto solves the key exchange problem but is 100× slower than AES. TLS uses asymmetric only to agree on a shared session key, then switches to AES for all data.

Forward Secrecy (ECDHE): Each session generates a new ephemeral key pair. The server's long-term private key is never used to encrypt data — only to authenticate. If the private key is stolen years later, past sessions cannot be decrypted.

SSL vs TLS History

graph LR
    classDef broken fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef deprecated fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef current fill:#3498db,stroke:#2471a3,color:#fff
    classDef preferred fill:#27ae60,stroke:#1e8449,color:#fff

    SSL2["SSL 2.0 (1995)<br/>BROKEN — DROWN attack<br/>Deprecated 1996"]:::broken --> SSL3
    SSL3["SSL 3.0 (1996)<br/>BROKEN — POODLE attack<br/>Deprecated 2015"]:::broken --> TLS10
    TLS10["TLS 1.0 (1999)<br/>BEAST, POODLE variants<br/>Deprecated 2021"]:::deprecated --> TLS11
    TLS11["TLS 1.1 (2006)<br/>Deprecated 2021"]:::deprecated --> TLS12
    TLS12["TLS 1.2 (2008)<br/>Minimum acceptable today<br/>2 RTT handshake"]:::current --> TLS13
    TLS13["TLS 1.3 (2018)<br/>CURRENT PREFERRED<br/>1 RTT, forward secrecy mandatory"]:::preferred

Minimum acceptable today: TLS 1.2. Prefer TLS 1.3.

TLS 1.2 Handshake (2 RTT)

sequenceDiagram
    participant C as Client
    participant S as Server

    rect rgb(40, 55, 75)
    Note over C,S: Round Trip 1 — negotiate + server proves identity
    C->>S: ClientHello<br/>TLS version: 1.2<br/>Random: 32 bytes<br/>Cipher suites: [TLS_ECDHE_RSA_AES256_GCM_SHA384, ...]<br/>Extensions: SNI=google.com, ALPN=h2

    S->>C: ServerHello<br/>Chosen cipher: TLS_ECDHE_RSA_AES256_GCM_SHA384<br/>Random: 32 bytes
    S->>C: Certificate<br/>*.google.com cert + intermediate CA chain
    S->>C: ServerKeyExchange<br/>ECDHE public key (ephemeral)<br/>Signed with server private key
    S->>C: ServerHelloDone
    end

    rect rgb(65, 50, 30)
    Note over C,S: Round Trip 2 — client verifies, both derive keys
    C->>C: Verify cert chain against OS root store
    C->>C: Generate pre-master secret from ECDHE keys
    C->>C: Derive session keys (AES key + MAC key)
    C->>S: ClientKeyExchange (ECDHE public key)
    C->>S: ChangeCipherSpec (switch to encryption)
    C->>S: Finished (HMAC of all handshake messages)

    S->>S: Derive same session keys
    S->>C: ChangeCipherSpec
    S->>C: Finished
    end

    rect rgb(40, 60, 45)
    Note over C,S: Encrypted data flows — 2 RTTs spent before the first byte
    C->>S: HTTP GET / (encrypted with AES-256-GCM)
    end

TLS 1.3 Handshake (1 RTT)

sequenceDiagram
    participant C as Client
    participant S as Server

    rect rgb(40, 60, 45)
    Note over C,S: Single Round Trip — most of the handshake, encrypted
    C->>S: ClientHello<br/>TLS 1.3<br/>key_share: ECDHE public key (X25519)<br/>supported_groups: X25519, P-256<br/>SNI: google.com<br/>ALPN: h2

    Note over S: Server derives keys NOW from client's key_share
    S->>C: ServerHello + key_share (server ECDHE public key)
    S->>C: EncryptedExtensions (ALPN negotiated)
    S->>C: Certificate (now ENCRYPTED — leaks no metadata)
    S->>C: CertificateVerify (signature over handshake transcript)
    S->>C: Finished (HMAC of transcript)

    Note over C: Client derives same keys, verifies cert, sends Finished
    C->>S: Finished
    C->>S: HTTP GET / (already encrypted — sent with Finished!)
    end

    Note over C,S: 1 RTT total — first encrypted byte leaves on flight #2

    rect rgb(65, 50, 30)
    Note over C,S: 0-RTT Resumption (optional, replay risk)
    C->>S: ClientHello + early_data (HTTP GET immediately)
    Note over S: No handshake needed — use pre-shared session ticket
    S->>C: HTTP Response
    end

TLS 1.3 vs TLS 1.2:

TLS Certificate Chain

graph TD
    classDef root fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef inter fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef leaf fill:#27ae60,stroke:#1e8449,color:#fff
    classDef endpoint fill:#3498db,stroke:#2471a3,color:#fff

    ROOT["Root CA Certificate<br/>DigiCert Global Root G2<br/>Self-signed<br/>Pre-installed in OS/browsers<br/>Private key stored OFFLINE (air-gapped HSM)"]:::root
    INTER["Intermediate CA Certificate<br/>DigiCert TLS RSA SHA256 2020 CA1<br/>Signed by Root CA<br/>Used for day-to-day signing"]:::inter
    LEAF["Leaf Certificate<br/>*.google.com<br/>Public key for TLS<br/>SANs: google.com, www.google.com<br/>Valid: 2024-01-01 to 2025-01-01<br/>Signed by Intermediate CA"]:::leaf

    ROOT -->|"signs"| INTER
    INTER -->|"signs"| LEAF
    LEAF -->|"presented during TLS handshake"| SERVER["google.com"]:::endpoint

Certificate fields:

Subject:    CN=*.google.com
Issuer:     CN=DigiCert TLS RSA SHA256 2020 CA1
SANs:       DNS:*.google.com, DNS:google.com
Not Before: 2024-01-15 00:00:00 UTC
Not After:  2025-02-14 23:59:59 UTC
Public Key: EC 256-bit (P-256)
Signature:  SHA256WithRSA

Why 3 levels? Root CA private keys are air-gapped — never online. Intermediate CA does daily signing. If an intermediate is compromised, revoke it without touching the root. The root change would require every OS/browser to update their trust store.

Certificate Validation Steps

flowchart TD
    classDef step fill:#3498db,stroke:#2471a3,color:#fff
    classDef ok fill:#27ae60,stroke:#1e8449,color:#fff
    classDef bad fill:#e74c3c,stroke:#c0392b,color:#fff

    RECV["Client receives certificate"]:::step --> SIG
    SIG["Verify signature chain<br/>Intermediate signed Leaf?<br/>Root signed Intermediate?"]:::step --> TRUST
    TRUST["Root CA in OS trust store?<br/>/etc/ssl/certs/ or system keychain"]:::step --> EXPIRY
    EXPIRY["Not Before &lt;= now &lt;= Not After?"]:::step --> SAN
    SAN["SAN matches requested hostname?<br/>*.google.com matches google.com?"]:::step --> REVOKE
    REVOKE["Not revoked?<br/>CRL or OCSP check"]:::step --> OK["Certificate VALID<br/>proceed with handshake"]:::ok
    SIG & TRUST & EXPIRY & SAN & REVOKE -->|"any fail"| ERR["TLS handshake FAILED<br/>connection aborted"]:::bad

Debugging TLS

# Full TLS handshake inspection
openssl s_client -connect google.com:443 -servername google.com
# Shows: cipher, cert chain, handshake details

# Check TLS version and cipher
openssl s_client -connect google.com:443 -tls1_3 2>/dev/null | grep "Protocol\|Cipher"

# Certificate expiry check
openssl s_client -connect google.com:443 2>/dev/null | openssl x509 -noout -dates

# Check all SANs on a certificate
openssl s_client -connect google.com:443 2>/dev/null | openssl x509 -noout -text | grep -A2 "Subject Alternative"

# Test specific TLS version (should fail for 1.0/1.1 on modern servers)
openssl s_client -connect google.com:443 -tls1   # TLS 1.0 — should fail
openssl s_client -connect google.com:443 -tls1_2 # TLS 1.2 — should work
openssl s_client -connect google.com:443 -tls1_3 # TLS 1.3 — should work

# curl with verbose TLS info
curl -v --tlsv1.3 https://google.com 2>&1 | grep -i "TLS\|SSL\|cert\|cipher"

Diffie-Hellman Key Exchange — The Mathematics

Diffie-Hellman solves a fundamental problem: two parties can agree on a shared secret over a public channel without ever transmitting the secret itself. Anyone eavesdropping sees only public values and cannot derive the secret.

Classic DH (Finite Field)

Public parameters (known to everyone, including attackers):

p = a large prime number (e.g., 2048-bit)
g = a generator (primitive root mod p, typically g=2 or g=5)

The exchange:

sequenceDiagram
    participant A as Alice (Client)
    participant NET as Public Network (attacker can see)
    participant B as Bob (Server)

    Note over A,B: Public params agreed in advance: p=23, g=5

    rect rgb(40, 55, 75)
    Note over A: Private computation — never leaves Alice
    A->>A: Pick secret a=6 (never transmitted)
    A->>A: Compute A = g^a mod p = 5^6 mod 23 = 8
    end
    A->>NET: Send A=8 (public)
    NET->>B: A=8

    rect rgb(40, 55, 75)
    Note over B: Private computation — never leaves Bob
    B->>B: Pick secret b=15 (never transmitted)
    B->>B: Compute B = g^b mod p = 5^15 mod 23 = 19
    end
    B->>NET: Send B=19 (public)
    NET->>A: B=19

    rect rgb(40, 60, 45)
    Note over A,B: Both derive the same secret independently
    A->>A: Shared secret = B^a mod p = 19^6 mod 23 = 2
    B->>B: Shared secret = A^b mod p = 8^15 mod 23 = 2
    Note over A,B: Both computed the same secret = 2
    end

    rect rgb(75, 40, 40)
    Note over NET: Attacker sees p=23, g=5, A=8, B=19<br/>Cannot find secret without solving discrete logarithm
    end

Why it works — the math:

Alice computes: s = B^a mod p = (g^b mod p)^a mod p = g^(ab) mod p
Bob computes:   s = A^b mod p = (g^a mod p)^b mod p = g^(ab) mod p

Both get g^(ab) mod p — the shared secret.
Attacker has A = g^a mod p and B = g^b mod p.
To find a from A = g^a mod p → Discrete Logarithm Problem.
No efficient algorithm known for large p (2048+ bits).

The Discrete Logarithm Problem: Given g, p, and A = g^a mod p, find a. Easy in one direction (exponentiation: fast), computationally infeasible in reverse for large primes.

ECDH — Elliptic Curve Diffie-Hellman (Used in TLS 1.3)

Classic DH uses modular arithmetic. ECDH uses points on an elliptic curve — same mathematical structure, but achieves equivalent security with much smaller keys.

Elliptic curve equation: y² = x³ + ax + b  (over a finite field)

Why elliptic curves?

Smaller keys → faster computation, smaller TLS handshake messages, less CPU.

Point multiplication on a curve:

graph LR
    classDef gen fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef pub fill:#3498db,stroke:#2471a3,color:#fff
    classDef secret fill:#27ae60,stroke:#1e8449,color:#fff

    G["Generator point G<br/>(public, on the curve)"]:::gen -->|"scalar multiplication"| PUB_A["Alice's public key<br/>A = a × G<br/>(a = Alice's private key scalar)"]:::pub
    G -->|"scalar multiplication"| PUB_B["Bob's public key<br/>B = b × G<br/>(b = Bob's private key scalar)"]:::pub
    PUB_A & PUB_B -->|"key agreement"| SECRET["Shared secret<br/>S = a × B = b × A = ab × G"]:::secret

The math:

Public curve parameters: curve equation + generator point G + field order n

Alice: pick private key a (random integer)
       compute public key A = a × G  (point multiplication on curve)

Bob:   pick private key b (random integer)
       compute public key B = b × G

Alice: shared secret = a × B = a × (b × G) = ab × G
Bob:   shared secret = b × A = b × (a × G) = ab × G

Both get the same point ab × G.
Attacker sees G, A, B. Must find a from A = a × G → Elliptic Curve Discrete Log Problem (ECDLP).
No efficient algorithm known.

Point multiplication (a × G) means: add point G to itself a times using the curve's addition law. Repeated doubling makes this O(log a) — fast. Reversing it (finding a given G and a × G) has no known polynomial-time algorithm.

ECDHE — Ephemeral ECDH (what TLS 1.3 actually uses)

ECDHE = ECDH + Ephemeral. A new key pair is generated for every TLS session.

sequenceDiagram
    participant C2 as Client
    participant S2 as Server

    Note over C2,S2: TLS 1.3 ECDHE with X25519 curve

    rect rgb(40, 55, 75)
    Note over C2,S2: Generate ephemeral key pairs — fresh every session
    C2->>C2: Generate ephemeral private key c_priv (random, 32 bytes)
    C2->>C2: Compute public key c_pub = c_priv × G
    C2->>S2: ClientHello: key_share = c_pub

    S2->>S2: Generate ephemeral private key s_priv (random, 32 bytes)
    S2->>S2: Compute public key s_pub = s_priv × G
    S2->>S2: Shared secret = s_priv × c_pub = s_priv × c_priv × G
    S2->>C2: ServerHello: key_share = s_pub

    C2->>C2: Shared secret = c_priv × s_pub = c_priv × s_priv × G
    end

    rect rgb(40, 60, 45)
    Note over C2,S2: Both derived same shared secret
    Note over C2,S2: Derive session keys via HKDF:
    Note over C2,S2: client_key = HKDF(shared_secret, "client key")
    Note over C2,S2: server_key = HKDF(shared_secret, "server key")
    end

    rect rgb(75, 40, 40)
    Note over C2: c_priv discarded after handshake
    Note over S2: s_priv discarded after handshake
    Note over C2,S2: Forward secrecy: past sessions unrecoverable
    end

Why "ephemeral" = forward secrecy:

• Static DH: server uses the same private key for all sessions. Steal the private key → decrypt all past recorded traffic.
• Ephemeral DH: server generates a new private key per session and discards it after the handshake. Steal the long-term private key → can only impersonate future connections, cannot decrypt past traffic. Each session's key material is gone forever.

X25519 — The Curve Used in TLS 1.3

TLS 1.3 mandates support for X25519 (Curve25519). Designed by Daniel J. Bernstein.

Curve equation: y² = x³ + 486662x² + x  (over the field of integers mod 2^255 - 19)
Base point G: u = 9
Field size: 2^255 - 19 (a Mersenne-like prime, chosen for fast arithmetic)
Key size: 256 bits
Security level: ~128 bits

Why X25519 over P-256 (NIST)?

From Shared Secret to Session Keys (HKDF)

The raw ECDHE output (a point on the curve) is not directly used as an AES key. It goes through HKDF (HMAC-based Key Derivation Function):

graph LR
    classDef input fill:#34495e,stroke:#212f3c,color:#fff
    classDef process fill:#f39c12,stroke:#ba6018,color:#fff
    classDef output fill:#27ae60,stroke:#1e8449,color:#fff

    ECDHE_OUT["ECDHE shared secret<br/>(32 bytes, the x-coordinate of ab×G)"]:::input --> HKDF_EXT
    HKDF_EXT["HKDF-Extract<br/>salt + IKM --> PRK<br/>(Pseudorandom Key)"]:::process --> HKDF_EXP
    TRANSCRIPT["Handshake transcript hash<br/>(all messages so far, prevents replay)"]:::input --> HKDF_EXP
    HKDF_EXP["HKDF-Expand<br/>PRK + label + context --> OKM<br/>(Output Key Material)"]:::process --> KEYS
    KEYS["client_write_key (AES-256)<br/>server_write_key (AES-256)<br/>client_write_IV (96-bit nonce)<br/>server_write_IV (96-bit nonce)"]:::output

The transcript hash binds the keys to this specific handshake — prevents man-in-the-middle attacks where an attacker replays a valid key exchange from a different session.

Summary of TLS 1.3 key derivation:

ECDHE_secret = s_priv × c_pub  (x-coordinate of the curve point)
early_secret = HKDF-Extract(0, PSK or 0)
handshake_secret = HKDF-Extract(derived(early_secret), ECDHE_secret)
master_secret = HKDF-Extract(derived(handshake_secret), 0)

client_handshake_traffic_secret = HKDF-Expand-Label(handshake_secret, "c hs traffic", transcript_hash)
server_handshake_traffic_secret = HKDF-Expand-Label(handshake_secret, "s hs traffic", transcript_hash)

mTLS — Mutual TLS Between Services

Standard TLS: client verifies server's certificate. mTLS: both sides present and verify certificates. The server also authenticates the client. This is the foundation of zero-trust service-to-service communication.

mTLS handshake vs TLS handshake

sequenceDiagram
    participant Client
    participant Server
    Client->>Server: ClientHello
    Server->>Client: Certificate (server proves its identity)
    Client->>Client: Verify server certificate chain
    Client->>Server: Key exchange, Finished
    Note over Client,Server: Connection established
    Note over Client,Server: Server identity verified — client remains anonymous

sequenceDiagram
    participant Client
    participant Server
    Client->>Server: ClientHello
    Server->>Client: Certificate
    Server->>Client: CertificateRequest (server asks for a client cert too)
    Client->>Server: Certificate (client proves its identity)
    Client->>Server: CertificateVerify
    Server->>Server: Verify client certificate against its own CA
    Note over Client,Server: Connection established
    Note over Client,Server: Both sides authenticated

cert-manager — automated certificate lifecycle

cert-manager is a Kubernetes controller that automates issuing, renewing, and rotating TLS certificates from multiple sources (Let's Encrypt, Vault, AWS PCA, self-signed CA).

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

# Verify
kubectl get pods -n cert-manager

Self-signed CA for internal service mTLS:

# Step 1: Create a self-signed CA certificate
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

---
# Step 2: Issue a CA certificate from the self-signed issuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io

---
# Step 3: Create a CA issuer that uses this CA to sign service certs
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-secret   # references the CA cert Secret above

Issue a service certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-tls
  namespace: payments
spec:
  secretName: payments-tls-secret    # cert-manager creates this Secret
  duration: 24h                      # short-lived = more secure
  renewBefore: 8h                    # renew when 8h remain
  dnsNames:
    - payments-svc.payments.svc.cluster.local
    - payments-svc.payments.svc
    - payments-svc
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer

# Mount the cert in your pod
spec:
  volumes:
    - name: tls
      secret:
        secretName: payments-tls-secret  # cert-manager keeps this up to date
  containers:
    - name: payments
      volumeMounts:
        - name: tls
          mountPath: /etc/tls
          readOnly: true
      env:
        - name: TLS_CERT_FILE
          value: /etc/tls/tls.crt
        - name: TLS_KEY_FILE
          value: /etc/tls/tls.key
        - name: CA_CERT_FILE
          value: /etc/tls/ca.crt

Implementing mTLS in a Go service

import (
    "crypto/tls"
    "crypto/x509"
    "os"
)

func newMTLSServer(certFile, keyFile, caFile string) (*http.Server, error) {
    // Load our own certificate and key
    cert, err := tls.LoadX509KeyPair(certFile, keyFile)
    if err != nil {
        return nil, err
    }

    // Load the CA that we use to verify client certificates
    caCert, err := os.ReadFile(caFile)
    if err != nil {
        return nil, err
    }
    caPool := x509.NewCertPool()
    caPool.AppendCertsFromPEM(caCert)

    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{cert},
        ClientAuth:   tls.RequireAndVerifyClientCert,  // enforce mTLS
        ClientCAs:    caPool,
        MinVersion:   tls.VersionTLS13,
    }

    return &http.Server{
        Addr:      ":8443",
        TLSConfig: tlsConfig,
    }, nil
}

func newMTLSClient(certFile, keyFile, caFile string) (*http.Client, error) {
    cert, _ := tls.LoadX509KeyPair(certFile, keyFile)
    caCert, _ := os.ReadFile(caFile)
    caPool := x509.NewCertPool()
    caPool.AppendCertsFromPEM(caCert)

    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{cert},  // present client cert
        RootCAs:      caPool,                    // verify server cert
        MinVersion:   tls.VersionTLS13,
    }
    return &http.Client{
        Transport: &http.Transport{TLSClientConfig: tlsConfig},
    }, nil
}

Certificate rotation — zero-downtime

cert-manager renews before expiry and updates the Secret. But pods that mounted the Secret at startup have the old cert in memory — they need to reload without restarting.

Option 1 — watch for file changes:

// Use inotify/fsnotify to reload certs when Secret is updated
watcher, _ := fsnotify.NewWatcher()
watcher.Add("/etc/tls/tls.crt")
go func() {
    for event := range watcher.Events {
        if event.Op&fsnotify.Write != 0 {
            newCert, _ := tls.LoadX509KeyPair(certFile, keyFile)
            tlsConfig.Certificates = []tls.Certificate{newCert}
            log.Println("TLS certificate rotated")
        }
    }
}()

Option 2 — use GetCertificate callback (preferred for servers):

tlsConfig := &tls.Config{
    GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
        // Called on every new TLS handshake — always reads the latest cert
        cert, err := tls.LoadX509KeyPair(certFile, keyFile)
        return &cert, err
    },
}
// Existing connections use the old cert; new connections use the new cert.
// No restart required.

Option 3 — Istio/Linkerd handle rotation transparently: Service meshes manage cert issuance and rotation for you. Sidecars hold the mTLS identity; your app code makes plain HTTP/gRPC calls; the sidecar wraps them in mTLS. Rotation is invisible to the app.

Let's Encrypt with cert-manager (public services)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: ops@myorg.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:                        # HTTP-01 challenge (port 80 reachable)
          ingress:
            class: nginx
      # Alternative: DNS-01 challenge (for wildcard certs / internal clusters)
      - dns01:
          route53:
            region: us-east-1
            hostedZoneID: Z123456789
            role: arn:aws:iam::123456789:role/cert-manager-dns

---
# Use the issuer in an Ingress annotation
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts: [api.example.com]
      secretName: api-tls-cert    # cert-manager auto-creates and renews this
  rules:
    - host: api.example.com
      ...

Debugging TLS/mTLS issues

# Test TLS connection and inspect certificate chain
openssl s_client -connect payments-svc:8443 \
  -servername payments-svc.payments.svc.cluster.local \
  -showcerts 2>/dev/null | openssl x509 -noout -text

# Test mTLS (present client cert)
openssl s_client -connect payments-svc:8443 \
  -cert /etc/tls/tls.crt \
  -key /etc/tls/tls.key \
  -CAfile /etc/tls/ca.crt

# Check cert expiry
openssl x509 -in /etc/tls/tls.crt -noout -dates
# notBefore=Jun  1 00:00:00 2026 GMT
# notAfter=Jun  2 00:00:00 2026 GMT  ← short-lived, 24h

# Check cert-manager certificate status
kubectl describe certificate payments-tls -n payments
# Conditions:
#   Ready: True   ← cert issued and valid
#   Ready: False  reason: Failed  ← look at Events for CA/DNS errors

# Check cert-manager logs for failures
kubectl logs -n cert-manager \
  -l app.kubernetes.io/component=controller --tail=100

# List all certificates and their expiry
kubectl get certificates -A
# NAME           READY   SECRET               AGE
# payments-tls   True    payments-tls-secret  2d