Distributed Transactions

Patterns for maintaining consistency across services and datastores without a global ACID transaction.

1. The Dual-Write Problem

Writing to two systems (DB + message queue) in sequence — no atomic boundary spans both.

graph TD
    classDef app fill:#34495e,stroke:#212f3c,color:#fff
    classDef db fill:#2980b9,stroke:#1f618d,color:#fff
    classDef queue fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff

    APP["Application code<br/>writes to two systems<br/>in sequence, no shared transaction"]:::app

    subgraph W1["Write 1 — local ACID transaction"]
        DB["Database<br/>BEGIN ... COMMIT<br/>durable the moment it returns"]:::db
    end

    subgraph W2["Write 2 — separate network call, separate failure domain"]
        MQ["Message queue<br/>publish(event)"]:::queue
    end

    APP -->|"1 . write row"| DB
    APP -.->|"2 . publish event<br/>can still fail independently of step 1"| MQ

    DB -.-> CRASH{{"Process crashes or times out<br/>between step 1 and step 2?"}}:::fail
    CRASH -.->|"DB committed,<br/>publish never happened"| GAP1["Consumers silently<br/>miss the update"]:::fail
    CRASH -.->|"publish somehow raced ahead<br/>of a DB write that then failed"| GAP2["Event fired for a<br/>transaction that never happened"]:::fail

Failure scenarios:

The root cause: you cannot COMMIT a database transaction and publish a message atomically with two different systems. One of them will always go first.

2. Two-Phase Commit (2PC)

A distributed protocol where a coordinator drives participants through two phases to achieve atomic commit across multiple nodes.

Phases

Why it's blocking and fragile

• If the coordinator crashes after Phase 1 but before Phase 2, all participants are stuck in the prepared state — they have locks held and cannot proceed or roll back without the coordinator's decision.
• Recovery requires reading the coordinator's WAL or waiting for it to restart — this can block for minutes in production.
• Any participant failure during Phase 2 requires the coordinator to retry indefinitely (or a human to intervene).

Coordinator crash window

graph LR
    classDef normal fill:#3498db,stroke:#2471a3,color:#fff
    classDef crash fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef stuck fill:#f39c12,stroke:#ba6018,color:#fff

    A["Coordinator: PREPARE sent,<br/>all votes YES received"]:::normal --> B{{"Coordinator crashes<br/>before writing COMMIT to its log"}}:::crash
    B -.->|"no decision was ever recorded"| C["Participants: locks still held,<br/>vote already cast YES,<br/>cannot commit or roll back on their own"]:::stuck
    B -.->|"eventually"| D["Coordinator restarts,<br/>reads its own WAL for a decision"]:::normal
    C -.->|"blocked until"| D

This is the uncertainty window — participants hold locks on rows they've prepared but cannot release them because they don't know if the coordinator committed or rolled back.

3. Spanner & TrueTime — Solving 2PC's Blocking Problem With Time

2PC's failure mode above isn't a bug in some particular implementation — it's structural. The moment the coordinator's decision becomes the single source of truth, every participant is one crashed coordinator away from holding locks indefinitely. Google Spanner is the system that solved this exact problem in production, across datacenters, using a fundamentally different approach: instead of a locking protocol that blocks other transactions while waiting for a coordinator's decision, it uses time itself as the synchronization mechanism.

The core insight: an honest clock

A locking protocol like 2PC blocks because a participant genuinely cannot know the outcome without asking someone else. Spanner sidesteps that by giving every node a way to reason about "has enough real time passed" without asking anyone. That only works because Google built TrueTime — an API that doesn't return a single timestamp the way System.currentTimeMillis() does. It returns a bounded uncertainty interval: TT.now() gives back [earliest, latest], an honest admission that the true current time lies somewhere in that window, not a claimed exact instant.

That interval stays narrow (typically a few milliseconds, not the unbounded skew a normal NTP-synced server can silently accumulate) because Google backs it with GPS receivers and atomic clocks in every datacenter, cross-checked against each other continuously. The interval isn't a rounding convenience — it's the mechanism the entire protocol depends on being trustworthy.

Commit-wait: turning "probably enough time" into "certainly enough time"

When Spanner commits a transaction, it assigns it a timestamp s. But picking s isn't the hard part — making sure no other transaction can observe the effects of this one before real time has actually reached s is. Spanner enforces this with commit-wait: before releasing locks or making the commit visible, it waits out the remaining uncertainty — TT.now().latest - s — so that by the time anything becomes visible, every node in the system is guaranteed real time has passed s, not just likely passed it.

That's the whole trick. A single imprecise clock reading "it's probably time" isn't good enough to order transactions across datacenters. A bounded, honest interval that the system can wait out is — because waiting out a known bound converts uncertainty into certainty. This is what makes Spanner externally consistent (linearizable across the entire system, not just within one replica set): if transaction A commits before transaction B starts anywhere in the world, every observer sees A's effects before B's.

Why this is a genuinely different tradeoff than 2PC

2PC blocks on a coordinator decision — a logical, communication-bound dependency. A participant can be stuck for minutes if the coordinator is slow to restart, because there's no bound on how long a network partition or a crashed process takes to resolve. Spanner "blocks" too, briefly — but only on TrueTime's own uncertainty bound, a physical, hardware-bounded wait that's typically single-digit milliseconds. It never holds a lock across a network round-trip to another node the way 2PC's prepare and commit phases do; it holds a lock only across a short, fixed wait against its own local clock reading. Short and bounded beats short and blocking-on-someone-else, which is exactly the property 2PC lacks.

The dependency this creates

This approach only works because Google controls the hardware in every datacenter — GPS antennas and atomic clock references are a physical infrastructure investment, not a software trick any team can drop into an existing cluster. That's precisely why TrueTime-style consistency stayed a Google-specific capability for years. CockroachDB and YugabyteDB later built similar ideas on top of regular NTP instead, accepting a much larger and less certain uncertainty bound in exchange for not needing custom hardware — a different point on the same tradeoff curve, not a free lunch.

sequenceDiagram
    participant App as Client transaction
    participant SP as Spanner leader replica
    participant TT as TrueTime API
    participant R as Replicas, other datacenters

    App->>SP: Commit transaction
    SP->>TT: TT.now()
    TT-->>SP: interval earliest to latest, pick s = latest
    SP->>R: Replicate write at timestamp s via Paxos
    R-->>SP: Majority acknowledges
    Note over SP: Commit-wait — hold locks until TT.now().earliest passes s
    SP->>App: Commit visible, locks released

4. 2PC Sequence Diagrams

Happy Path

sequenceDiagram
    participant C as Coordinator
    participant P1 as Participant 1 (orders DB)
    participant P2 as Participant 2 (inventory DB)

    rect rgb(40, 55, 75)
    Note over C,P2: Phase 1 — Prepare (no data visible yet)
    C->>P1: PREPARE
    activate P1
    C->>P2: PREPARE
    activate P2
    P1-->>C: YES (WAL written, row locked)
    P2-->>C: YES (WAL written, row locked)
    end

    Note over C: Both voted YES — write COMMIT to<br/>coordinator's own log (point of no return)

    rect rgb(40, 60, 45)
    Note over C,P2: Phase 2 — Commit
    C->>P1: COMMIT
    C->>P2: COMMIT
    P1-->>C: ACK, locks released
    deactivate P1
    P2-->>C: ACK, locks released
    deactivate P2
    end

Coordinator Crashes After Prepare

sequenceDiagram
    participant C as Coordinator
    participant P1 as Participant 1 (orders DB)
    participant P2 as Participant 2 (inventory DB)

    rect rgb(40, 55, 75)
    Note over C,P2: Phase 1 — Prepare completes normally
    C->>P1: PREPARE
    activate P1
    C->>P2: PREPARE
    activate P2
    P1-->>C: YES
    P2-->>C: YES
    end

    rect rgb(65, 50, 30)
    Note over C: CRASH — before COMMIT is written to the log
    Note over P1: Holding locks, voted YES,<br/>uncertain whether the outcome is commit or rollback
    Note over P2: Holding locks, voted YES,<br/>uncertain whether the outcome is commit or rollback
    Note over P1,P2: Blocked — cannot unilaterally commit or roll back
    end

    Note over C: Coordinator restarts, reads its WAL,<br/>replays the decision to both participants
    C->>P1: COMMIT or ROLLBACK (from recovered log)
    C->>P2: COMMIT or ROLLBACK (from recovered log)
    deactivate P1
    deactivate P2

5. Three-Phase Commit (3PC)

Adds a pre-commit phase between Prepare and Commit to reduce the uncertainty window.

Phases: Prepare → Pre-Commit → Commit

What it adds: After receiving all YES votes, coordinator sends PRE-COMMIT. Participants acknowledge. Only then does the coordinator send COMMIT. If the coordinator crashes after PRE-COMMIT, participants can infer the coordinator intended to commit and proceed unilaterally after a timeout.

Still has issues:

• Network partitions can cause split-brain: some participants receive PRE-COMMIT, others don't. After a coordinator crash, different participants may make different decisions.
• Adds a full RTT of latency vs 2PC.
• Rarely used in practice because of the network partition problem — Saga or Outbox patterns are preferred.

6. Saga Pattern

A saga is a sequence of local transactions. Each step updates one service's database. On failure, compensating transactions undo the preceding steps.

Key property: No global lock. Each local transaction commits immediately and is visible. Compensation is semantic (business-level undo), not a database rollback.

Choreography

Services react to events without a central coordinator. Each service listens for events, does its local transaction, and emits the next event.

Pros:

• No single point of failure
• Services are fully decoupled
• Simple to add new steps by subscribing to events

Cons:

• Hard to track overall saga state (distributed observability problem)
• Cyclic dependencies between services are easy to create accidentally
• Testing the full flow requires all services running

Orchestration

A Saga Orchestrator (a service or workflow engine) explicitly calls each participant and drives the flow. It knows the full saga state.

Pros:

• Central place to observe, debug, and retry saga state
• Easier to reason about compensations
• Can use durable execution (Temporal, AWS Step Functions)

Cons:

• Orchestrator is a single point of coordination (not a SPOF if made durable, but a coupling point)
• Services must expose APIs the orchestrator calls — more coupling than events

7. Saga Choreography Diagram

Order → Inventory → Payment → Shipping, with compensation on failure.

sequenceDiagram
    participant Q as EventBus
    participant O as OrderService
    participant I as InventoryService
    participant P as PaymentService
    participant S as ShippingService

    rect rgb(40, 60, 45)
    Note over Q,S: Happy path — each service reacts to the previous event, does a local commit, emits the next
    O->>Q: OrderCreated
    Q->>I: OrderCreated
    I->>I: local commit — reserve items
    I->>Q: InventoryReserved
    Q->>P: InventoryReserved
    P->>P: local commit — charge card
    P->>Q: PaymentCharged
    Q->>S: PaymentCharged
    S->>S: local commit — create shipment
    S->>Q: ShipmentCreated
    end

    rect rgb(65, 50, 30)
    Note over Q,S: Payment fails — compensations run in reverse, each triggered by an event, not a direct call
    P->>Q: PaymentFailed
    Q->>I: PaymentFailed
    I->>I: compensate — release reservation
    I->>Q: InventoryReleased
    Q->>O: InventoryReleased
    O->>O: compensate — cancel order
    O->>Q: OrderCancelled
    end

Compensating transactions:

Try It: Live Saga Simulator

Same Order → Inventory → Payment → Shipping chain as the diagram above, except this one's live. Click through it step by step, then pick exactly when to fail — after 1 step, after 2, or all the way to the end with no failure at all. Everything already committed compensates automatically, backward, one step at a time, in reverse order — anything that hasn't run yet is never touched.

8. Outbox Pattern

Write to an outbox table in the same database transaction as your business data. A separate relay process reads the outbox and publishes to the message broker.

Why it works: The outbox write and the business write share the same ACID transaction. Either both commit or both roll back. The relay publishes only after the DB transaction is committed.

-- Outbox table
CREATE TABLE outbox (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_id TEXT NOT NULL,
    event_type  TEXT NOT NULL,
    payload     JSONB NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT now(),
    published_at TIMESTAMPTZ
);

-- Business transaction (atomic)
BEGIN;
  INSERT INTO orders (id, user_id, status) VALUES ($1, $2, 'pending');
  INSERT INTO outbox (aggregate_id, event_type, payload)
    VALUES ($1, 'OrderCreated', $3);
COMMIT;

The relay queries unpublished rows, publishes to the broker, then marks them published:

-- Relay: poll and publish
SELECT * FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100;
-- ... publish each to broker ...
UPDATE outbox SET published_at = now() WHERE id = $1;

At-least-once delivery: If the relay crashes after publishing but before marking the row, it will republish on restart. Consumers must be idempotent.

9. Outbox Sequence Diagram

sequenceDiagram
    participant App as Application
    participant DB as PostgreSQL
    participant Relay as OutboxRelay
    participant Broker as MessageBroker

    rect rgb(40, 60, 45)
    Note over App,DB: One atomic transaction — both rows commit or neither does
    App->>DB: BEGIN
    App->>DB: INSERT INTO orders
    App->>DB: INSERT INTO outbox (published_at = NULL)
    App->>DB: COMMIT
    DB-->>App: OK
    end

    loop Poll every N ms
        Relay->>DB: SELECT unpublished outbox rows
        DB-->>Relay: rows
        Relay->>Broker: publish event
        Broker-->>Relay: ACK
        Relay->>DB: UPDATE outbox SET published_at = now()
    end

    rect rgb(65, 50, 30)
    Note over Relay,Broker: Relay crashes after publish but before the UPDATE lands
    Relay->>Broker: publish event
    Broker-->>Relay: ACK
    Note over Relay: crash — published_at never gets set
    Note over Relay,DB: On restart, the row still shows unpublished →<br/>relay republishes it → consumer must dedupe
    end

10. Change Data Capture (CDC) with Debezium

CDC reads the database replication log (Postgres WAL, MySQL binlog) to capture every committed change as a stream of events. No polling, no outbox table needed in the application code.

graph LR
    classDef app fill:#34495e,stroke:#212f3c,color:#fff
    classDef pg fill:#2980b9,stroke:#1f618d,color:#fff
    classDef cdc fill:#e67e22,stroke:#ba6018,color:#fff
    classDef stream fill:#8e44ad,stroke:#6c3483,color:#fff

    APP["Application<br/>writes to Postgres normally<br/>(no outbox table, no dual write)"]:::app --> PG

    subgraph PGNODE["PostgreSQL — wal_level = logical"]
        PG["INSERT / UPDATE / DELETE<br/>on tracked tables"]:::pg --> WAL["WAL<br/>every committed change,<br/>durable fact"]:::pg
    end

    WAL -->|"replication slot protocol<br/>same as a streaming replica"| DBZ["Debezium connector<br/>tracks LSN, resumes exactly<br/>where it left off after restart"]:::cdc

    DBZ -->|"emits change event:<br/>before/after row image + LSN + tx metadata"| KAFKA["Kafka topic<br/>per tracked table"]:::stream
    DBZ -.->|"DDL changes"| SCHEMA["Schema history topic"]:::stream

How Debezium works with Postgres:

1. Postgres is configured with wal_level = logical.
2. Debezium connects as a replication slot reader — same protocol as a streaming replica.
3. Every committed INSERT/UPDATE/DELETE on tracked tables emits a change event to Kafka.
4. The event contains before and after row images, LSN (log sequence number), and transaction metadata.

Why it's reliable:

• Events come directly from the WAL — they are committed facts, not speculative writes.
• The replication slot tracks the LSN so Debezium can resume exactly where it left off after a restart.
• No dual-write: the application writes to DB normally; CDC is a side effect of the WAL.

Tradeoffs:

• Requires wal_level = logical (minor storage overhead for WAL retention).
• Schema changes (DDL) must be handled carefully — Debezium tracks schema history in a Kafka topic.
• Replication slots accumulate WAL if the consumer falls behind — can fill disk.

11. Idempotency Across Services

An idempotent operation produces the same result if called multiple times. Critical for at-least-once delivery systems.

Idempotency key flow:

1. Client generates a unique key (UUID or hash of request content) and sends it in the request header: Idempotency-Key: <uuid>.
2. Service checks a deduplication table before processing.
3. If key exists: return the stored response immediately, skip processing.
4. If key is new: insert the key, process, store the response.

CREATE TABLE idempotency_keys (
    key         TEXT PRIMARY KEY,
    response    JSONB NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT now(),
    expires_at  TIMESTAMPTZ
);

-- Before processing
SELECT response FROM idempotency_keys WHERE key = $1 AND expires_at > now();

-- After successful processing (in same transaction as business logic)
INSERT INTO idempotency_keys (key, response, expires_at)
VALUES ($1, $2, now() + interval '24 hours')
ON CONFLICT (key) DO NOTHING;

TTL: Keys can be expired after a safe window (24h–7d) once the upstream caller no longer retries.

12. Distributed Locking

Use when multiple instances must not execute a critical section simultaneously (e.g., scheduled job, inventory deduction).

Redis — SET NX PX

# Acquire: SET key value NX PX <ttl_ms>
SET lock:order:123 <owner_token> NX PX 5000

# Release: only if we own the lock (Lua for atomicity)
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
end

• NX: only set if not exists
• PX 5000: auto-expire after 5 seconds (prevents deadlock on crash)
• Always use a unique owner token and check it before releasing — prevents releasing someone else's lock

Redlock (multi-node Redis)

Acquire the lock on N/2+1 independent Redis nodes within a time window. If quorum is reached, the lock is held. Protects against single Redis node failure.

Controversy: Redlock is debated (Martin Kleppmann vs Antirez). In practice, for most use cases, a single Redis instance with NX PX is sufficient if you can tolerate the Redis instance being a SPOF.

ZooKeeper / etcd

• Use ephemeral nodes (ZooKeeper) or leases (etcd) — lock is automatically released if the holder crashes or disconnects.
• Stronger consistency guarantees than Redis (linearizable by default in etcd).
• Higher latency than Redis (~1–5ms vs ~0.1ms).

Optimistic vs Pessimistic Locking

13. Optimistic Concurrency Control

No locks. Each row carries a version field. The writer checks the version hasn't changed before committing.

-- Schema
ALTER TABLE orders ADD COLUMN version INT NOT NULL DEFAULT 0;

-- Read
SELECT id, status, version FROM orders WHERE id = $1;

-- Update (check-and-set)
UPDATE orders
SET status = 'paid', version = version + 1
WHERE id = $1 AND version = $2;
-- If 0 rows updated: someone else modified it — retry

HTTP ETags:

GET /orders/123
→ ETag: "v42"

PUT /orders/123
   If-Match: "v42"
→ 200 OK if version matches
→ 412 Precondition Failed if it was modified

ETags are the HTTP-native expression of optimistic concurrency — the ETag is the version, If-Match is the check-and-set.

14. TCC (Try-Confirm-Cancel)

A reservation pattern that avoids holding database locks across service boundaries.

Three phases per participant:

1. Try: Reserve resources tentatively (don't commit them). E.g., mark inventory as "reserved" but not "deducted".
2. Confirm: If all Tries succeeded, confirm all reservations (make them permanent).
3. Cancel: If any Try failed, cancel all reservations (release the tentative holds).

Example — inventory reservation:

Try:     UPDATE inventory SET reserved = reserved + 5 WHERE sku = 'X' AND available >= 5
Confirm: UPDATE inventory SET available = available - 5, reserved = reserved - 5 WHERE sku = 'X'
Cancel:  UPDATE inventory SET reserved = reserved - 5 WHERE sku = 'X'

How it avoids blocking:

• Resources are "reserved" not "locked". Other transactions can still read and reserve (if enough available).
• The Confirm/Cancel phase is fast — no coordination needed after Try.
• Resources don't stay in limbo: a timeout on unconfirmed reservations triggers automatic Cancel.

graph LR
    classDef start fill:#34495e,stroke:#212f3c,color:#fff
    classDef tryState fill:#f39c12,stroke:#ba6018,color:#fff
    classDef okState fill:#27ae60,stroke:#1e8449,color:#fff
    classDef cancelState fill:#e74c3c,stroke:#c0392b,color:#fff

    AVAIL["Available<br/>reserved = 0"]:::start -->|"Try<br/>reserved += 5, no lock held"| TRIED["Tried<br/>reserved = 5, available unchanged"]:::tryState

    TRIED -->|"Confirm — every participant's<br/>Try succeeded"| CONFIRMED["Confirmed<br/>available -= 5, reserved -= 5 (permanent)"]:::okState
    TRIED -->|"Cancel — a Try failed,<br/>or TTL expired unconfirmed"| CANCELLED["Cancelled<br/>reserved -= 5, available unchanged"]:::cancelState

    CANCELLED -.->|"stock is free again"| AVAIL

Tradeoffs:

• All participants must implement three separate endpoints/operations.
• Requires a timeout/cleanup mechanism for stuck Tries.
• The orchestrator must track which participants were successfully Tried.

15. Decision Table

16. Real-World Examples

E-Commerce Order (Saga Orchestration)

sequenceDiagram
    participant Orch as Saga Orchestrator
    participant Ord as OrderService
    participant Inv as InventoryService
    participant Pay as PaymentService
    participant Ship as ShippingService

    rect rgb(40, 60, 45)
    Note over Orch,Ship: Forward path — orchestrator calls each step explicitly, waits for local commit
    Orch->>Ord: 1. createOrder()
    Ord-->>Orch: local commit OK
    Orch->>Inv: 2. reserve()
    Inv-->>Orch: local commit OK
    Orch->>Pay: 3. charge()
    Pay-->>Orch: FAILS
    end

    rect rgb(65, 50, 30)
    Note over Orch,Ship: Compensation path — orchestrator walks completed steps backward
    Orch->>Inv: compensate step 2 — release()
    Inv-->>Orch: released
    Orch->>Ord: compensate step 1 — cancel()
    Ord-->>Orch: cancelled
    end

State machine stored in the orchestrator (e.g., DynamoDB or Postgres). Each step is idempotent. Retries are safe. Note that step 4 (ShippingService.createShipment()) never runs here — the orchestrator only calls forward as far as the failure, then walks backward through the steps that actually committed.

Payment Processing (2PC within DB, Saga across services)

Within a single Postgres cluster (e.g., debit account A, credit account B in the same DB):

• Use a regular DB transaction — this is exactly the use case 2PC was designed for at the DB level.
• The DB engine handles 2PC internally; you write BEGIN ... COMMIT.

Across services (payment gateway, ledger service, notification service):

• Use Saga Orchestration.
• Payment gateway call is the critical step — design it as idempotent with an idempotency key.
• On payment failure, compensate the ledger reservation.

Inventory Reservation for Flash Sale (TCC)

Try:     Reserve N units (add to reserved_count) — fast, low contention
         Return reservation_id

Confirm: Deduct from available_count, clear reservation
         Triggered when order is confirmed

Cancel:  Release reservation_count
         Triggered on: payment failure, TTL expiry, user abandonment

TTL on reservations (e.g., 10 minutes) prevents ghost reservations from blocking stock indefinitely. A background job sweeps expired reservations and triggers Cancel.