Async Patterns — Message Queues and Event-Driven Architecture

How services hand off work without blocking on each other — queue patterns, delivery guarantees, dead letter queues, the outbox and saga patterns for keeping data consistent across services, event sourcing, CQRS, backpressure, and idempotent consumers.

1. Why Async

Core benefits:

• Decouple producers and consumers — deploy/scale independently
• Absorb spikes — queue depth grows instead of requests failing
• Retry on failure — message stays in queue until acked
• Parallelism — multiple workers consume same queue

2. Sync vs Async Request Flow

sequenceDiagram
    participant User
    participant API
    participant Worker
    participant DB

    rect rgb(80, 30, 30)
    Note over User,DB: Sync flow — caller blocks for the entire duration of the work
    User->>API: POST /resize-image
    API->>Worker: process(image)
    Worker->>DB: save result
    DB-->>Worker: ok
    Worker-->>API: done
    API-->>User: 200 OK (waits 3s)
    end

    rect rgb(25, 55, 45)
    Note over User,DB: Async flow — caller is released the instant the job is queued
    User->>API: POST /resize-image
    API->>DB: enqueue job (status=pending)
    API-->>User: 202 Accepted + job_id
    Worker->>DB: poll queue
    Worker->>DB: save result (status=completed)
    User->>API: GET /jobs/job_id
    API-->>User: 200 completed
    end

3. Message Delivery Guarantees

Practical rule: build for at-least-once + idempotent consumers. Exactly-once is expensive and rarely worth it.

4. Queue Patterns

Point-to-Point

One message → one consumer. Workers compete; each message processed once.

graph LR
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff
    classDef skip fill:#7f8c8d,stroke:#616a6b,color:#fff

    P["Producer"]:::svc --> Q

    subgraph DELIVERY["Point-to-point — exactly one consumer sees each message"]
        Q["Queue<br/>one message, one consumer"]:::svc
        Q --> A["Consumer A<br/>receives and processes it"]:::svc
        Q -.->|"never delivered here"| B["Consumer B<br/>does not see this message"]:::skip
    end

Use: task queues, job workers. SQS standard queue.

Pub/Sub

One message → all subscribers. Fan-out.

graph LR
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff

    P["Producer"]:::svc --> T["Topic<br/>fan-out to every subscriber"]:::svc

    subgraph SUBS["Every subscriber gets its own independent copy"]
        A["Subscriber A"]:::svc
        B["Subscriber B"]:::svc
        C["Subscriber C"]:::svc
    end

    T --> A
    T --> B
    T --> C

Use: event broadcast, notifications. SNS topics, Kafka topics.

Competing Consumers

Multiple workers on the same queue for horizontal scale.

graph LR
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff
    classDef queue fill:#e67e22,stroke:#ba6018,color:#fff

    P["Producer"]:::svc --> Q["Queue<br/>shared work backlog"]:::queue

    subgraph POOL["Worker pool — horizontal scale, each message goes to exactly one worker"]
        W1["Worker 1"]:::svc
        W2["Worker 2"]:::svc
        W3["Worker 3"]:::svc
    end

    Q --> W1
    Q --> W2
    Q --> W3

SQS: workers call ReceiveMessage concurrently; visibility timeout prevents double-processing.

Priority Queue

High-priority messages processed before low-priority.

• RabbitMQ: x-max-priority queue arg, per-message priority property
• SQS FIFO: no native priority — workaround: separate queues per priority tier, consume high-priority queue first

# RabbitMQ priority queue declaration
channel.queue_declare(queue='jobs', arguments={'x-max-priority': 10})
channel.basic_publish(
    exchange='', routing_key='jobs',
    body='urgent task',
    properties=pika.BasicProperties(priority=9)
)

5. Dead Letter Queue (DLQ)

Messages go to DLQ when:

• Max receive count exceeded (SQS: maxReceiveCount)
• TTL expired (RabbitMQ: x-message-ttl)
• Consumer explicitly rejects without requeue

graph LR
    classDef queue fill:#3498db,stroke:#2471a3,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef action fill:#8e44ad,stroke:#6c3483,color:#fff

    Q["Queue"]:::queue -->|"Worker fails 3x<br/>maxReceiveCount exceeded"| DLQ

    subgraph RECOVERY["Recovery workflow — everything after a message is parked, all manual until replay"]
        DLQ["Dead Letter Queue"]:::fail --> ALERT["Alert on-call<br/>CloudWatch alarm"]:::action
        ALERT --> INSPECT["Inspect message +<br/>fix consumer bug"]:::action
        INSPECT --> REPLAY["Reprocess:<br/>replay back to source queue"]:::action
    end

    REPLAY -.->|"retried"| Q

SQS DLQ config:

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123:my-dlq",
    "maxReceiveCount": 3
  }
}

Monitoring:

• CloudWatch alarm on ApproximateNumberOfMessagesVisible on DLQ > 0
• Alert immediately — DLQ message = data not processed

Reprocessing: replay DLQ messages back to source queue after fixing the consumer bug.

6. Outbox Pattern

Problem: dual-write — you need to write to DB and publish to queue atomically. If the app crashes between the two, you get inconsistency.

Solution: write event to an outbox table in the same DB transaction. Separate poller reads outbox and publishes.

graph TD
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff
    classDef db fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef poller fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef queue fill:#e67e22,stroke:#ba6018,color:#fff

    SVC["Service"]:::svc --> TXN

    subgraph TXN["Single DB transaction — atomic"]
        ORD["INSERT INTO orders (...)"]:::db
        OUT["INSERT INTO outbox<br/>(event_type, payload, published=false)"]:::db
        ORD --> OUT
    end

    subgraph RELAY["Async relay — polls continuously, independent of the writing transaction"]
        POLL["Poller<br/>SELECT * FROM outbox WHERE published=false"]:::poller
        PUB["Publish to queue"]:::queue
        MARK["UPDATE outbox SET published=true"]:::poller
        POLL --> PUB --> MARK
    end

    OUT --> POLL

CREATE TABLE outbox (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_type  TEXT NOT NULL,
    payload     JSONB NOT NULL,
    published   BOOLEAN DEFAULT false,
    created_at  TIMESTAMPTZ DEFAULT now()
);

Tools that implement this: Debezium (CDC-based), Transactional Outbox libraries.

Guarantee: at-least-once delivery. Poller must handle crash between publish and mark-published → idempotent consumers required.

7. Saga Pattern

Distributed transactions across microservices without 2PC. Each step has a compensation transaction to undo on failure.

graph LR
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff
    subgraph HAPPY["Happy path — each service reacts only to the event before it"]
        Order["OrderService"]:::svc -->|"order.created"| Inv["InventoryService"]:::svc
        Inv -->|"inventory.reserved"| Pay["PaymentService"]:::svc
        Pay -->|"payment.charged"| Ship["ShippingService"]:::svc
        Ship -->|"order.shipped"| Done["Order complete"]:::svc
    end
    subgraph COMPENSATE["Compensation chain — triggered only by payment.failed"]
        InvC["InventoryService"]:::fail
        OrderC["OrderService"]:::fail
        Cancel["Order cancelled"]:::fail
    end
    Pay -.->|"payment.failed"| InvC
    InvC -.->|"inventory.released"| OrderC
    OrderC -.->|"order.cancelled"| Cancel

sequenceDiagram
    participant O as SagaOrchestrator
    participant Inv as InventoryService
    participant Pay as PaymentService
    participant Ship as ShippingService
    rect rgb(25, 55, 45)
    Note over O,Pay: Happy path so far — each step commits before the next begins
    O->>Inv: 1. reserve(order_id)
    Inv-->>O: reserved_ok
    O->>Pay: 2. charge(order_id)
    Pay-->>O: charge_failed
    end
    rect rgb(80, 30, 30)
    Note over O,Inv: On failure at step N, compensate steps 1..N-1
    O->>Inv: compensate: release(order_id)
    Inv-->>O: released_ok
    end

8. Saga Sequence — E-Commerce Order

sequenceDiagram
    participant Orchestrator
    participant Inventory
    participant Payment
    participant Shipping

    rect rgb(25, 55, 45)
    Note over Orchestrator,Payment: Happy path — each step commits before the next begins
    Orchestrator->>Inventory: reserve_items(order_id)
    Inventory-->>Orchestrator: reserved_ok
    Orchestrator->>Payment: charge_card(order_id)
    Payment-->>Orchestrator: charge_failed
    end

    rect rgb(80, 30, 30)
    Note over Orchestrator,Inventory: Compensation — undo everything that already succeeded
    Orchestrator->>Inventory: release_items(order_id)
    Inventory-->>Orchestrator: released_ok
    Orchestrator-->>Orchestrator: mark_order_failed(order_id)
    end

9. Event Sourcing

State = current value (traditional) vs State = sequence of events (event sourcing).

graph TD
    classDef trad fill:#3498db,stroke:#2471a3,color:#fff
    classDef event fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef replay fill:#e67e22,stroke:#ba6018,color:#fff

    subgraph Traditional["Traditional — state = current value"]
        ROW["orders table<br/>{ id: 1, status: 'shipped', total: 99 }"]:::trad
    end

    subgraph EventSourced["Event sourced — state = sequence of events"]
        E1["OrderCreated<br/>{ id: 1, total: 99 }"]:::event
        E2["PaymentCharged<br/>{ id: 1, amount: 99 }"]:::event
        E3["OrderShipped<br/>{ id: 1, tracking: 'XYZ' }"]:::event
        E1 --> E2 --> E3
        E3 --> R["Replay events →<br/>current state"]:::replay
    end

Benefits:

• Full audit log for free
• Rebuild read projections at any point in time
• Debug by replaying history

Tradeoffs:

• Querying current state requires replay or a projection
• Schema evolution of old events is hard
• Store grows forever (use snapshots)

Snapshot pattern: periodically store current state so replay starts from snapshot, not event 0.

10. CQRS

Separate write model (commands) from read model (queries).

graph TD
    classDef client fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef write fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef read fill:#27ae60,stroke:#1e8449,color:#fff

    C["Client"]:::client -->|"Command"| WAPI["Write API"]:::write
    C -->|"Query"| RAPI["Read API"]:::read

    subgraph WriteSide["Write model — consistent"]
        WAPI --> WDB["Write DB<br/>normalized (Postgres)"]:::write
    end

    subgraph ReadSide["Read model — fast, denormalized"]
        RAPI --> RDB["Read DB<br/>denormalized (Elasticsearch/Redis)"]:::read
    end

    WDB -->|"async projection"| RDB

Write DB: optimized for consistency (Postgres, normalized). Read DB: optimized for query patterns (Elasticsearch, Redis, denormalized Postgres view).

Eventual consistency gap: read model lags write model by milliseconds to seconds. Client must handle stale reads (show "processing" state).

CQRS + Event Sourcing pair naturally: events from write side populate read projections.

11. Backpressure

Consumer falls behind → queue grows → memory/disk exhaustion → cascade failure.

graph TD
    classDef bad fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef strat fill:#3498db,stroke:#2471a3,color:#fff
    classDef trigger fill:#7f8c8d,stroke:#616a6b,color:#fff

    C["Consumer falls behind"]:::trigger --> G["Queue depth grows"]:::trigger

    subgraph UNCHECKED["Left unchecked"]
        M["Memory/disk exhaustion"]:::bad --> F["Cascade failure"]:::bad
    end

    G --> M

    subgraph MITIGATE["Mitigation strategies — pick based on whether losing data is acceptable"]
        S1["Drop<br/>discard new messages"]:::strat
        S2["Buffer<br/>short in-memory burst absorption"]:::strat
        S3["Block producer<br/>producer waits<br/>Kafka lag, TCP flow control"]:::strat
        S4["Rate limit<br/>reject excess with 429"]:::strat
        S5["Scale consumers<br/>add workers, KEDA"]:::strat
    end

    G --> S1
    G --> S2
    G --> S3
    G --> S4
    G --> S5

KEDA + SQS autoscale:

triggers:
- type: aws-sqs-queue
  metadata:
    queueURL: https://sqs.us-east-1.amazonaws.com/123/my-queue
    queueLength: "10"       # scale up when >10 messages per replica
    awsRegion: us-east-1

12. Idempotency

At-least-once delivery = consumers will see duplicate messages. Consumers must be idempotent.

Idempotency key pattern:

sequenceDiagram
    participant P as Producer
    participant C as Consumer
    participant T as dedup_table

    P->>C: message { idempotency_key: "order-123-payment", ... }
    C->>T: SELECT WHERE idempotency_key = 'order-123-payment'
    alt key found
        T-->>C: row exists
        C-->>C: skip — already processed
    else key not found
        T-->>C: no row
        C->>C: process message
        C->>T: INSERT idempotency_key (same transaction)
    end

CREATE TABLE processed_events (
    idempotency_key TEXT PRIMARY KEY,
    processed_at    TIMESTAMPTZ DEFAULT now()
);

-- In consumer (single transaction):
INSERT INTO processed_events (idempotency_key) VALUES ($1)
ON CONFLICT DO NOTHING
RETURNING idempotency_key;
-- Only process if row was inserted (not a duplicate)

SQS deduplication: FIFO queues have built-in 5-minute dedup window using MessageDeduplicationId.

13. SQS Specifics

Visibility Timeout

After ReceiveMessage, message hidden from other consumers for VisibilityTimeout seconds.

• Worker must DeleteMessage before timeout or message reappears
• Set timeout > max processing time
• Extend with ChangeMessageVisibility for long jobs

sequenceDiagram
    participant W1 as Worker
    participant Q as Queue
    participant W2 as Other worker

    W1->>Q: ReceiveMessage
    Q-->>W1: message (invisible for VisibilityTimeout, e.g. 30s)
    alt Worker finishes in time
        W1->>Q: DeleteMessage
        Note over Q: message gone for good
    else Worker crashes or is too slow
        Note over Q: timeout expires
        Q-->>W2: message visible again
        W2->>Q: retries processing
    end

Long Polling

WaitTimeSeconds: 20 — connection held open until message arrives or 20s elapses. Reduces empty receives and cost.

sqs.receive_message(
    QueueUrl=queue_url,
    WaitTimeSeconds=20,
    MaxNumberOfMessages=10
)

FIFO vs Standard

Message Groups (FIFO)

MessageGroupId = partition key. Messages in same group are strictly ordered. Different groups processed in parallel.

sqs.send_message(
    QueueUrl=fifo_url,
    MessageBody=json.dumps(event),
    MessageGroupId=f"user-{user_id}",
    MessageDeduplicationId=event_id
)

Delay Queues

DelaySeconds (0-900): message invisible on arrival. Use for retry backoff, scheduled tasks.

14. Kafka vs SQS vs RabbitMQ

Choose Kafka when: replay needed, high throughput, event sourcing, stream processing (Kafka Streams, Flink). Choose SQS when: AWS-native, simple task queue, zero ops. Choose RabbitMQ when: complex routing, low latency, existing AMQP ecosystem.

15. Real Examples

Order Processing Pipeline

graph TD
    classDef api fill:#3498db,stroke:#2471a3,color:#fff
    classDef svc fill:#27ae60,stroke:#1e8449,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff

    subgraph INGEST["Ingest"]
        API["POST /orders<br/>validates + persists (status=pending)"]:::api --> EVT["publish to order-events<br/>Kafka/SNS"]:::api
    end

    subgraph FANOUT["Fan-out — three independent subscribers, same event"]
        INV["inventory-service<br/>reserve stock"]:::svc
        PAY["payment-service<br/>charge card"]:::svc
        NOTIF["notification-service<br/>send confirmation email"]:::svc
    end

    EVT --> INV
    EVT --> PAY
    EVT --> NOTIF

    subgraph SAGA["Saga compensation — only runs on payment failure"]
        COMP["saga compensates:<br/>inventory-service releases stock"]:::fail --> FAILED["order status → failed"]:::fail
    end

    PAY -.->|"payment.failed"| COMP

Image Resize Pipeline

graph TD
    classDef api fill:#3498db,stroke:#2471a3,color:#fff
    classDef worker fill:#27ae60,stroke:#1e8449,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff

    subgraph UPLOAD["Upload"]
        U["Upload API"]:::api -->|"PutObject"| S3["S3"]:::api
        S3 -->|"S3 Event"| Q["SQS queue"]:::api
    end

    subgraph PROCESS["Worker — one job per uploaded image"]
        W["Worker ECS/Lambda<br/>bucket, key, sizes: thumb/medium/large"]:::worker
        DL["download original"]:::worker
        RS["resize to each size"]:::worker
        UP["upload resized to S3"]:::worker
        DM["DeleteMessage"]:::worker
        W --> DL --> RS --> UP --> DM
    end

    Q --> W
    W -.->|"corrupt/unsupported format"| DLQ["DLQ"]:::fail
    Q -.->|"scale on depth, KEDA"| W

Notification Service (Fan-out)

graph LR
    classDef svc fill:#3498db,stroke:#2471a3,color:#fff
    classDef worker fill:#27ae60,stroke:#1e8449,color:#fff

    ANY["Any service"]:::svc --> SNS["SNS topic<br/>user.notifications"]:::svc

    subgraph CHANNELS["Delivery channels — each dedups independently on notification_id + channel"]
        EQ["SQS queue<br/>email-worker"]:::worker --> SES["sends via SES"]:::worker
        PQ["SQS queue<br/>push-worker"]:::worker --> FCM["sends via FCM/APNs"]:::worker
        SQ["SQS queue<br/>sms-worker"]:::worker --> TW["sends via Twilio"]:::worker
    end

    SNS --> EQ
    SNS --> PQ
    SNS --> SQ

Each worker:

• idempotency key = notification_id + channel
• dedup before sending to avoid double-send on retry

Audit Log (Event Sourcing)

graph TD
    classDef api fill:#3498db,stroke:#2471a3,color:#fff
    classDef kafka fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef store fill:#7f8c8d,stroke:#616a6b,color:#fff

    API["Every write API"]:::api -->|"publishes AuditEvent"| KAFKA["Kafka audit-log topic<br/>retention: 90 days, compliance"]:::kafka

    subgraph SINKS["Durable sinks — same event, two purposes"]
        PG["Postgres audit_events table<br/>append-only"]:::store
        ES["Elasticsearch index<br/>search"]:::store
    end

    KAFKA --> PG
    KAFKA --> ES
    PG -.->|"filter by resource_id"| REPLAY["Replay: rebuild full<br/>resource history"]:::store

AuditEvent: {
  event_id, timestamp, user_id, action,
  resource_type, resource_id, before, after
}

Quick Reference

Decouple + absorb spikes          → queue (SQS/Kafka)
Broadcast to many consumers       → pub/sub (SNS, Kafka topics)
Guaranteed processing + retry     → at-least-once + DLQ
Atomic DB write + publish         → outbox pattern
Distributed transaction           → saga (choreography or orchestration)
Full history + time travel        → event sourcing
Fast reads, separate write model  → CQRS
Prevent duplicate processing      → idempotency key + dedup table
Consumer falling behind           → backpressure (drop/block/scale/rate-limit)