Saga Pattern: Distributed Transactions Without 2PC

The saga pattern is a way to manage distributed transactions in microservices by breaking one large operation into a sequence of smaller local transactions — each with a compensating transaction that can undo it if something fails downstream.

You're building an e-commerce checkout. It needs to: reserve inventory, charge the payment, and create a shipment — in different services, each owning its own database. Payment succeeds. Shipment creation fails. You've charged the user for something you can't ship.

In a monolith with a single database, a transaction handles this — all steps succeed or all roll back together. In microservices, you can't do a single atomic transaction across separate databases. That's the core problem the saga pattern solves.

A saga sequences local transactions across services. Each service completes its step and emits an event (or the next step is called directly). If any step fails, the saga runs compensating transactions in reverse — undoing completed steps one by one. No distributed lock. No coordinator waiting for two-phase acknowledgment. Just a chain of local commits with defined rollback paths.

This post covers the full picture: how sagas work, choreography vs orchestration with real code, how to design compensating transactions, a complete Kafka implementation, how sagas compare to 2PC, and how to debug them in production.

Why Distributed Transactions Are Hard

A traditional database transaction is ACID — Atomic, Consistent, Isolated, Durable. When a transaction touches one database, the engine can lock rows, guarantee ordering, and roll back atomically on failure.

In microservices, each service has its own database. You can't lock rows across PostgreSQL, MongoDB, and MySQL in a single transaction. The common alternative is two-phase commit (2PC): a coordinator asks all participants to prepare, then commits if all say yes, or aborts if anyone says no.

2PC works but it has serious problems at scale:

Blocking: All participants hold locks while waiting for the coordinator's final commit message.
Coordinator failure: If the coordinator crashes after sending "prepare" but before sending "commit", participants hang indefinitely — they can't unilaterally decide.
Latency: Every write needs two round trips across the network before it's committed.

For high-throughput microservices, 2PC is a bottleneck. Most teams avoid it. The saga pattern is the practical alternative.

What Is a Saga? (Exact Mechanics)

A saga replaces one distributed transaction with N local transactions plus N compensating transactions.

plaintext

Checkout Saga — Happy Path:
1. Reserve inventory      → success → emit InventoryReserved
2. Charge payment         → success → emit PaymentCharged
3. Create shipment        → success → emit OrderFulfilled
 
Checkout Saga — Failure at Step 3:
1. Reserve inventory      → success (committed)
2. Charge payment         → success (committed)
3. Create shipment        → FAILS
 
Compensation (run in reverse):
3. (failed, nothing to undo)
2. Refund payment         ← compensating transaction
1. Release inventory      ← compensating transaction

Compensating transactions are not rollbacks in the database sense. They're forward-moving operations — new writes that reverse the effect of a previous step. A refund is not an undo of a charge; it's a new credit applied to the account.

This distinction matters for design. Each saga step must be designed with its compensation in mind from the start.

Choreography vs Orchestration Sagas

There are two ways to implement the saga pattern. The choice affects how services communicate, where business logic lives, and how easy it is to debug.

Choreography: Event-Driven, No Central Coordinator

In choreography, each service listens for events and reacts. Services don't call each other directly — they emit events to a message bus (Kafka, RabbitMQ) and other services consume those events.

plaintext

Event flow (happy path):
Order Service       → publishes "OrderCreated"
Inventory Service   → receives "OrderCreated" → reserves stock → publishes "InventoryReserved"
Payment Service     → receives "InventoryReserved" → charges card → publishes "PaymentCharged"
Shipping Service    → receives "PaymentCharged" → creates shipment → publishes "OrderFulfilled"
 
Event flow (payment failure):
Payment Service     → charge fails → publishes "PaymentFailed"
Inventory Service   → receives "PaymentFailed" → releases stock → publishes "InventoryReleased"
Order Service       → receives "InventoryReleased" → marks order as failed

Each service only knows about events — not about other services. This gives you strong decoupling.

python

# Inventory service — Kafka consumer/producer
from kafka import KafkaConsumer, KafkaProducer
import json
import logging
 
logger = logging.getLogger(__name__)
 
consumer = KafkaConsumer(
    "order-events",
    group_id="inventory-service",
    bootstrap_servers="kafka:9092",
    value_deserializer=lambda m: json.loads(m.decode("utf-8")),
    enable_auto_commit=False,
)
producer = KafkaProducer(
    bootstrap_servers="kafka:9092",
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
)
 
for message in consumer:
    event = message.value
 
    if event["type"] == "OrderCreated":
        order_id = event["order_id"]
        items = event["items"]
 
        if reserve_stock(order_id, items):
            producer.send("inventory-events", {
                "type": "InventoryReserved",
                "order_id": order_id,
            })
            logger.info(f"[{order_id}] Inventory reserved")
        else:
            producer.send("inventory-events", {
                "type": "InventoryReservationFailed",
                "order_id": order_id,
                "reason": "Out of stock",
            })
            logger.warning(f"[{order_id}] Inventory reservation failed")
 
    elif event["type"] == "PaymentFailed":
        order_id = event["order_id"]
        release_stock(order_id)  # Compensating transaction
        producer.send("inventory-events", {
            "type": "InventoryReleased",
            "order_id": order_id,
        })
        logger.info(f"[{order_id}] Inventory released (compensation)")
 
    consumer.commit()

Pros of choreography:

Services are truly independent — you can deploy them separately.
Adding a new step means adding a new consumer, not touching existing services.
No single coordinator to fail.

Cons of choreography:

The overall saga flow is implicit — it lives in the combination of all event handlers, not in one place.
Debugging requires tracing events across multiple services and multiple Kafka topics.
Cyclic dependencies can emerge if event subscriptions aren't carefully designed.
Hard to understand what the "full transaction" looks like without reading every service.

Orchestration: Central Coordinator Controls the Flow

In orchestration, a saga orchestrator (a dedicated service or workflow engine) explicitly calls each service in order and handles failures.

python

# Checkout Saga Orchestrator — Python with async/await
import asyncio
from dataclasses import dataclass, field
from enum import Enum
from datetime import datetime
from typing import List
 
 
class SagaStep(str, Enum):
    STARTED = "STARTED"
    RESERVING_INVENTORY = "RESERVING_INVENTORY"
    CHARGING_PAYMENT = "CHARGING_PAYMENT"
    CREATING_SHIPMENT = "CREATING_SHIPMENT"
    COMPLETED = "COMPLETED"
    COMPENSATING = "COMPENSATING"
    FAILED = "FAILED"
 
 
@dataclass
class SagaState:
    saga_id: str
    order_id: str
    step: SagaStep = SagaStep.STARTED
    completed_steps: List[str] = field(default_factory=list)
    error: str = ""
    created_at: datetime = field(default_factory=datetime.utcnow)
    updated_at: datetime = field(default_factory=datetime.utcnow)
 
 
class CheckoutSaga:
    def __init__(self, db, order_id: str):
        self.db = db
        self.order_id = order_id
        self.state = SagaState(
            saga_id=f"saga-{order_id}",
            order_id=order_id,
        )
 
    async def execute(self, order_data: dict):
        await self._persist_state()
 
        # Step 1: Reserve inventory
        await self._transition(SagaStep.RESERVING_INVENTORY)
        try:
            await inventory_service.reserve(self.order_id, order_data["items"])
            self.state.completed_steps.append("inventory_reserved")
            await self._persist_state()
        except Exception as e:
            return await self._compensate(error=f"Inventory failed: {e}")
 
        # Step 2: Charge payment
        await self._transition(SagaStep.CHARGING_PAYMENT)
        try:
            await payment_service.charge(self.order_id, order_data["amount"])
            self.state.completed_steps.append("payment_charged")
            await self._persist_state()
        except Exception as e:
            return await self._compensate(error=f"Payment failed: {e}")
 
        # Step 3: Create shipment
        await self._transition(SagaStep.CREATING_SHIPMENT)
        try:
            await shipping_service.create(self.order_id, order_data["address"])
            self.state.completed_steps.append("shipment_created")
            await self._persist_state()
        except Exception as e:
            return await self._compensate(error=f"Shipping failed: {e}")
 
        await self._transition(SagaStep.COMPLETED)
        return {"success": True, "saga_id": self.state.saga_id}
 
    async def _compensate(self, error: str):
        self.state.step = SagaStep.COMPENSATING
        self.state.error = error
        await self._persist_state()
 
        # Undo in reverse order of completion
        if "payment_charged" in self.state.completed_steps:
            await payment_service.refund(self.order_id)
 
        if "inventory_reserved" in self.state.completed_steps:
            await inventory_service.release(self.order_id)
 
        await self._transition(SagaStep.FAILED)
        return {"success": False, "error": error, "saga_id": self.state.saga_id}
 
    async def _transition(self, step: SagaStep):
        self.state.step = step
        self.state.updated_at = datetime.utcnow()
        await self._persist_state()
 
    async def _persist_state(self):
        # Upsert saga state to database before every step
        await self.db.upsert("saga_states", self.state.__dict__)

Pros of orchestration:

The complete saga flow is visible in one place.
Failures and compensations are explicit and traceable.
Easier to add observability — the orchestrator logs every state transition.

Cons of orchestration:

The orchestrator is a potential single point of failure (mitigated by making it stateless and using persistent state).
Services become somewhat coupled to the orchestrator's interface.
The orchestrator can turn into a "God service" if not carefully bounded.

Choosing Between Them

	Choreography	Orchestration
Flow visibility	Distributed across services	Centralized in orchestrator
Service coupling	Low (event contracts only)	Moderate (orchestrator calls services)
Debugging	Hard — trace events across topics	Easier — single state machine
Best for	Simple flows, many services	Complex flows, clear business logic
Risk	Cyclic event dependencies	Orchestrator as bottleneck

For most teams, start with orchestration. The visibility wins pay off in debugging and iteration speed. Move to choreography when you genuinely need services to be independently deployable without any shared orchestrator dependency.

Compensating Transactions

Compensating transactions are the mechanism that makes sagas safe. Design them carefully — they need to handle edge cases that rarely happen in practice but will happen in production.

Design Principles

1. Compensations must be idempotent.

If the saga retries compensation (due to a crash or network timeout), running the compensation twice must produce the same result as running it once.

python

# BAD: Not idempotent — double compensation doubles the refund
def refund_payment(order_id, amount):
    stripe.refund.create(amount=amount)
 
# GOOD: Idempotent — checks if refund already exists
def refund_payment(order_id, amount):
    existing = db.query(Refund).filter_by(order_id=order_id).first()
    if existing:
        return existing  # Already compensated — skip
    refund = stripe.refund.create(
        amount=amount,
        idempotency_key=f"refund-{order_id}",  # Stripe deduplication
    )
    db.add(Refund(order_id=order_id, refund_id=refund.id))
    db.commit()
    return refund

2. Compensations can fail too.

What happens when your compensation fails? A refund attempt might fail because the payment provider is down. You need a strategy:

Retry with backoff: Keep retrying the compensation until it succeeds. Store failed compensations in a queue.
Dead letter queue: After N retries, route to a dead letter queue for manual handling.
Alert and escalate: Some compensation failures require human intervention (e.g., charge succeeded, refund keeps failing — someone needs to manually process the refund).

python

import asyncio
 
async def compensate_with_retry(compensation_fn, max_retries=5, backoff_base=2):
    for attempt in range(max_retries):
        try:
            await compensation_fn()
            return
        except Exception as e:
            if attempt == max_retries - 1:
                # Send to dead letter queue for manual handling
                await dead_letter_queue.send({
                    "compensation": compensation_fn.__name__,
                    "error": str(e),
                    "order_id": compensation_fn.order_id,
                })
                raise CompensationPermanentlyFailed(str(e))
            wait = backoff_base ** attempt
            await asyncio.sleep(wait)

3. Some operations can't be compensated.

Sending an email, publishing a push notification, or triggering a third-party webhook — these can't be undone. Design strategies:

Move irreversible steps to the end. If sending the confirmation email is the last step, and it fails, you can treat that as a partial success (the order is fulfilled, just resend the email).
Accept the side effect. For some operations, sending a "we're sorry, your order was cancelled" email after compensation is the right answer.
Use outbox pattern. Don't send the email directly inside the saga. Write to an outbox table, and let a separate process send it only after the saga commits fully.

Implementing Saga with a Message Queue (Kafka)

Here's a complete, production-relevant Kafka implementation for an order processing saga. This shows the orchestration pattern where the orchestrator uses Kafka for async communication with services.

python

# saga_orchestrator.py — Order saga with Kafka
from kafka import KafkaConsumer, KafkaProducer
from kafka.errors import KafkaError
import json
import uuid
import logging
from enum import Enum
 
logger = logging.getLogger(__name__)
 
COMMAND_TOPIC = "saga-commands"
REPLY_TOPIC = "saga-replies"
 
 
class SagaOrchestrator:
    """
    Sends commands to services via Kafka.
    Receives replies from services via REPLY_TOPIC.
    Persists state to DB before each step.
    """
 
    def __init__(self, db):
        self.db = db
        self.producer = KafkaProducer(
            bootstrap_servers="kafka:9092",
            value_serializer=lambda v: json.dumps(v).encode("utf-8"),
        )
 
    def start_checkout_saga(self, order_id: str, order_data: dict) -> str:
        saga_id = str(uuid.uuid4())
        self._save_state(saga_id, order_id, "RESERVING_INVENTORY", [])
 
        # First command — orchestrator kicks off the saga
        self.producer.send(COMMAND_TOPIC, {
            "saga_id": saga_id,
            "command": "RESERVE_INVENTORY",
            "order_id": order_id,
            "items": order_data["items"],
            "reply_to": REPLY_TOPIC,
        })
        logger.info(f"[{saga_id}] Saga started — sent RESERVE_INVENTORY")
        return saga_id
 
    def handle_reply(self, reply: dict):
        saga_id = reply["saga_id"]
        state = self._load_state(saga_id)
 
        if state is None:
            logger.error(f"[{saga_id}] No state found for saga")
            return
 
        event_type = reply["type"]
        logger.info(f"[{saga_id}] Received reply: {event_type}")
 
        if event_type == "INVENTORY_RESERVED":
            self._update_state(saga_id, "CHARGING_PAYMENT",
                               state["completed_steps"] + ["inventory_reserved"])
            self.producer.send(COMMAND_TOPIC, {
                "saga_id": saga_id,
                "command": "CHARGE_PAYMENT",
                "order_id": state["order_id"],
                "amount": reply["amount"],
                "reply_to": REPLY_TOPIC,
            })
 
        elif event_type == "PAYMENT_CHARGED":
            self._update_state(saga_id, "CREATING_SHIPMENT",
                               state["completed_steps"] + ["payment_charged"])
            self.producer.send(COMMAND_TOPIC, {
                "saga_id": saga_id,
                "command": "CREATE_SHIPMENT",
                "order_id": state["order_id"],
                "address": reply["address"],
                "reply_to": REPLY_TOPIC,
            })
 
        elif event_type == "SHIPMENT_CREATED":
            self._update_state(saga_id, "COMPLETED",
                               state["completed_steps"] + ["shipment_created"])
            logger.info(f"[{saga_id}] Saga completed successfully")
 
        # Failure paths → trigger compensation
        elif event_type in ("INVENTORY_RESERVATION_FAILED", "PAYMENT_FAILED",
                            "SHIPMENT_CREATION_FAILED"):
            self._compensate(saga_id, state, reason=event_type)
 
    def _compensate(self, saga_id: str, state: dict, reason: str):
        self._update_state(saga_id, "COMPENSATING", state["completed_steps"])
        completed = state["completed_steps"]
 
        if "payment_charged" in completed:
            self.producer.send(COMMAND_TOPIC, {
                "saga_id": saga_id,
                "command": "REFUND_PAYMENT",
                "order_id": state["order_id"],
                "reply_to": REPLY_TOPIC,
            })
 
        if "inventory_reserved" in completed:
            self.producer.send(COMMAND_TOPIC, {
                "saga_id": saga_id,
                "command": "RELEASE_INVENTORY",
                "order_id": state["order_id"],
                "reply_to": REPLY_TOPIC,
            })
 
        self._update_state(saga_id, "FAILED", completed, error=reason)
        logger.warning(f"[{saga_id}] Saga compensated. Reason: {reason}")
 
    def _save_state(self, saga_id, order_id, step, completed_steps, error=""):
        self.db.execute(
            "INSERT INTO saga_states (saga_id, order_id, step, completed_steps, error) "
            "VALUES (%s, %s, %s, %s, %s)",
            (saga_id, order_id, step, json.dumps(completed_steps), error)
        )
        self.db.commit()
 
    def _load_state(self, saga_id):
        row = self.db.execute(
            "SELECT * FROM saga_states WHERE saga_id = %s", (saga_id,)
        ).fetchone()
        if row:
            return {**row, "completed_steps": json.loads(row["completed_steps"])}
        return None
 
    def _update_state(self, saga_id, step, completed_steps, error=""):
        self.db.execute(
            "UPDATE saga_states SET step=%s, completed_steps=%s, error=%s "
            "WHERE saga_id=%s",
            (step, json.dumps(completed_steps), error, saga_id)
        )
        self.db.commit()
 
 
# Reply consumer — runs in a separate thread/process
def run_reply_consumer(orchestrator: SagaOrchestrator):
    consumer = KafkaConsumer(
        REPLY_TOPIC,
        group_id="saga-orchestrator",
        bootstrap_servers="kafka:9092",
        value_deserializer=lambda m: json.loads(m.decode("utf-8")),
        enable_auto_commit=False,
    )
    for message in consumer:
        try:
            orchestrator.handle_reply(message.value)
            consumer.commit()
        except Exception as e:
            logger.error(f"Failed to handle reply: {e}")
            # Don't commit — Kafka will re-deliver

The key insight: the orchestrator never holds locks. It sends a command, records state, and exits. When a reply arrives, it reads its persisted state and decides the next step. If the orchestrator crashes between sending a command and receiving the reply, it can recover by reading the database and continuing from the last committed step.

Saga vs Two-Phase Commit (2PC)

	Saga	Two-Phase Commit (2PC)
Consistency model	Eventual consistency	Strong (ACID) consistency
Lock behavior	No distributed locks	Participants hold locks until coordinator commits
Performance	Fast — no cross-service blocking	Slow — 2 network round trips per write
Failure handling	Compensating transactions	Coordinator failure can block all participants
Partial failure	Handled explicitly by compensation	Handled by coordinator abort
Scalability	Good — each service operates independently	Poor — locks become bottlenecks at scale
Operational complexity	Medium — need to design compensations	High — 2PC coordinators are hard to operate
Suitable for	Microservices with independent DBs	Legacy distributed DBs, single-vendor setups
Data visibility	Intermediate states are visible	Participants see committed state only

The critical difference is what happens during a failure. In 2PC, if the coordinator crashes after sending "prepare" but before "commit", all participants sit holding locks until the coordinator recovers. In a saga, each service commits immediately and the orchestrator drives recovery through compensation — no participant is ever blocked waiting for a global decision.

When 2PC is the right answer: If you control the entire stack (e.g., multiple tables in PostgreSQL using its built-in two-phase commit), 2PC gives you strong consistency with manageable overhead. The problems arise at the distributed systems level where coordinators and participants are separate processes on separate machines.

When to choose saga: Any microservices architecture where services own independent databases. Especially: order processing, user onboarding flows, travel booking, multi-step financial workflows.

Debugging Sagas in Production

Sagas are harder to debug than monolithic transactions because the state is distributed across services and events. Here's what works.

1. Saga State Table as the Source of Truth

The saga state table is your first debug tool. Query it to see exactly where a saga is:

sql

-- Find all sagas stuck in COMPENSATING for more than 10 minutes
SELECT saga_id, order_id, step, error, updated_at
FROM saga_states
WHERE step IN ('COMPENSATING', 'RUNNING')
  AND updated_at < NOW() - INTERVAL '10 minutes';
 
-- Full history for a specific order
SELECT * FROM saga_states
WHERE order_id = 'order-abc123'
ORDER BY created_at;

2. Structured Logging with Correlation IDs

Every log line from every service should include saga_id and order_id. This lets you grep across services.

python

import structlog
 
log = structlog.get_logger().bind(
    saga_id=saga_id,
    order_id=order_id,
    service="inventory-service",
)
 
log.info("inventory.reserve.start", items=items)
log.info("inventory.reserve.success", reserved_count=len(items))
log.error("inventory.reserve.failed", reason="out_of_stock")

With structured logs, a single query in Datadog, Loki, or CloudWatch gives you the full timeline across all services for one saga.

3. Distributed Tracing

Propagate trace context (OpenTelemetry) through every Kafka message and HTTP call. Your saga becomes a single trace with spans per service.

python

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
 
# When producing a Kafka message — inject trace context into headers
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("saga.charge_payment"):
    headers = {}
    inject(headers)  # Adds traceparent, tracestate headers
    producer.send("saga-commands", value=command, headers=list(headers.items()))
 
# When consuming — extract context from headers
context = extract(dict(message.headers))
with tracer.start_as_current_span("payment.charge", context=context):
    do_charge()

4. Workflow Engines (Temporal, AWS Step Functions)

For complex sagas, consider using a dedicated workflow engine instead of hand-rolling orchestration.

Temporal (open-source): Write your saga as a Go or Python workflow function. Temporal handles persistence, retries, timeouts, and compensation automatically. The saga code looks like regular sequential code but runs durably across failures.

python

# Temporal workflow — saga as a durable function
from temporalio import workflow, activity
 
@workflow.defn
class CheckoutSagaWorkflow:
    @workflow.run
    async def run(self, order_data: dict) -> dict:
        try:
            await workflow.execute_activity(
                reserve_inventory,
                order_data,
                start_to_close_timeout=timedelta(seconds=30),
            )
            await workflow.execute_activity(
                charge_payment,
                order_data,
                start_to_close_timeout=timedelta(seconds=30),
            )
            await workflow.execute_activity(
                create_shipment,
                order_data,
                start_to_close_timeout=timedelta(seconds=30),
            )
            return {"status": "completed"}
        except Exception as e:
            # Temporal handles retries; on final failure, run compensation
            await workflow.execute_activity(refund_payment, order_data)
            await workflow.execute_activity(release_inventory, order_data)
            return {"status": "failed", "error": str(e)}

Temporal's UI shows the full execution history, current state, pending activities, and retry counts — all in one place.

AWS Step Functions: Managed state machine service. Define your saga as a JSON/YAML state machine. Step Functions handles retries, error catching, and branching. The execution history is stored in AWS and queryable via the console or API. Good fit if you're already on AWS and don't want to operate Temporal yourself.

5. Saga Monitoring Metrics

Track these metrics to catch problems early:

saga_started_total — count of sagas initiated
saga_completed_total — successful completions
saga_compensation_triggered_total — how often compensations run (by reason)
saga_step_duration_seconds — latency per step (p50, p95, p99)
saga_stuck_count — sagas in a non-terminal state for > N minutes

Alert on: compensation rate spike, stuck sagas, and step latency above SLA.

Keeping Track of Saga State (Crash Recovery)

The orchestrator must persist state before every step transition. If it crashes mid-saga, the recovery process reads the database and resumes.

python

# Saga state schema
class SagaState(Base):
    __tablename__ = "saga_states"
 
    saga_id = Column(String, primary_key=True)
    order_id = Column(String, index=True)
    step = Column(String)           # "RESERVING_INVENTORY", "COMPENSATING", etc.
    status = Column(String)         # "RUNNING", "COMPLETED", "FAILED"
    completed_steps = Column(JSON)  # ["inventory_reserved", "payment_charged"]
    error = Column(Text, nullable=True)
    created_at = Column(DateTime, default=datetime.utcnow)
    updated_at = Column(DateTime, onupdate=datetime.utcnow)

Recovery worker (run on startup or periodically):

python

async def recover_stuck_sagas(db):
    # Find sagas that were running when the orchestrator crashed
    stuck = db.query(SagaState).filter(
        SagaState.status == "RUNNING",
        SagaState.updated_at < datetime.utcnow() - timedelta(minutes=5)
    ).all()
 
    for saga in stuck:
        logger.warning(f"Recovering stuck saga {saga.saga_id} at step {saga.step}")
        # Resume from last known state
        await resume_saga(saga)

When to Use Sagas

Good fit:

Multi-step business workflows spanning multiple services (order processing, user onboarding, travel booking)
When you need reliability without 2PC overhead
When eventual consistency is acceptable for the business domain

Not a good fit:

Simple two-service coordination (a direct synchronous call is fine)
High-frequency, low-latency operations where saga overhead matters
When you genuinely need immediate consistency (financial ledger balance checks)
When your "microservices" could reasonably be a single service — don't add saga complexity for complexity's sake

Key Takeaways

A saga is a sequence of local transactions with compensating transactions for each step — no distributed lock needed
Choreography: services react to events — loose coupling, distributed flow, hard to trace
Orchestration: central coordinator drives steps — visible flow, easier to debug, slight coupling
Compensating transactions must be idempotent — they may run more than once
Some operations (emails, webhooks) can't be compensated — put them last, or use the outbox pattern
Persist saga state before every step — this enables crash recovery
Use Temporal or Step Functions to avoid hand-rolling orchestration for complex flows
Sagas give you eventual consistency, not immediate consistency — know when that's acceptable

FAQ

What is the saga pattern in microservices?

The saga pattern is a way to manage long-running distributed transactions across multiple microservices without using a two-phase commit protocol. Instead of one atomic operation, a saga chains together local transactions — each service commits its own change, and if any step fails, compensating transactions undo the previous steps in reverse order.

What is the difference between choreography and orchestration in sagas?

In choreography, services communicate through events on a message bus. Each service listens for relevant events and reacts, with no central coordinator. In orchestration, a dedicated saga orchestrator sends commands to each service and receives replies, explicitly managing the flow and compensation. Choreography offers looser coupling; orchestration offers better visibility and debuggability.

What is a compensating transaction?

A compensating transaction is the "undo" operation for a saga step. If step 3 fails after steps 1 and 2 have already committed, the saga runs compensating transactions for steps 2 and 1 in reverse order. Compensations are forward-moving operations (a refund, a stock release) not database rollbacks. They must be idempotent — safe to run multiple times.

When should I use saga pattern vs two-phase commit?

Use the saga pattern when services own separate databases and you can't coordinate with a shared transaction. Sagas give you eventual consistency with good scalability. Use 2PC when you control a single database system that supports it natively (e.g., multi-table transactions in PostgreSQL) and need strong consistency. Avoid 2PC in distributed microservices — coordinator failures cause blocking, and cross-service latency makes it slow.

How do I handle saga failures in production?

First, make all compensating transactions idempotent and retry them with exponential backoff. Persist saga state to a database before every step so you can recover after crashes. Route permanently-failed compensations to a dead letter queue for manual handling. Set up alerts on stuck sagas (in a non-terminal state past a timeout) and on compensation rate spikes. Use distributed tracing to reconstruct the full timeline across services.

What tools can I use to implement sagas?

For message-based sagas: Kafka (topic-per-event) or RabbitMQ (direct exchanges). For orchestration engines: Temporal (open-source, self-hosted or cloud) or AWS Step Functions (managed). For state persistence: any relational database works well — PostgreSQL is common. For observability: OpenTelemetry for distributed tracing, structured logging with correlation IDs, and Prometheus/Grafana for saga metrics.

How does the saga pattern handle partial failures?

When any step fails, the saga transitions to a compensating state and runs compensating transactions for all previously completed steps, in reverse order. Only steps that actually completed need compensation. The saga tracks which steps completed via a completed_steps list persisted to a database. If the orchestrator itself fails mid-compensation, the recovery worker reads the persisted state and resumes compensation from the last known point.

Saga Pattern: Distributed Transactions Without 2PC

Why Distributed Transactions Are Hard

What Is a Saga? (Exact Mechanics)

Choreography vs Orchestration Sagas

Choreography: Event-Driven, No Central Coordinator

Orchestration: Central Coordinator Controls the Flow

Choosing Between Them

Compensating Transactions

Design Principles

Implementing Saga with a Message Queue (Kafka)

Saga vs Two-Phase Commit (2PC)

Debugging Sagas in Production

1. Saga State Table as the Source of Truth

2. Structured Logging with Correlation IDs

3. Distributed Tracing

4. Workflow Engines (Temporal, AWS Step Functions)

5. Saga Monitoring Metrics

Keeping Track of Saga State (Crash Recovery)

When to Use Sagas

Key Takeaways

FAQ

Enjoyed this article?

Related Posts

Message Queues Explained: Kafka vs RabbitMQ (2026 Guide)

ACID Properties Explained: Database Transactions

API Gateway Pattern: The Front Door to Your Microservices