Circuit Breaker Pattern: What It Is, How It Works, and When to Use It

The circuit breaker pattern is a fault-tolerance design pattern that stops a distributed system from repeatedly calling a failing service, instead returning errors immediately to prevent cascading failures. Named after the electrical circuit breaker in your home, it trips when too many failures occur — cutting off traffic to the broken service so the rest of your system keeps running.

Here is why it matters in practice: your payment service goes down. Without a circuit breaker, every checkout request waits 30 seconds for a timeout before failing. Threads pile up, the checkout service runs out of resources, then the API gateway starts failing. One broken service took down everything. With a circuit breaker, after five failures the breaker trips — subsequent requests fail in milliseconds, not 30 seconds. Your checkout page returns a useful error immediately, other features keep working, and the moment payment recovers the breaker resets automatically.

How Circuit Breakers Work

Think of a circuit breaker exactly like the one in your house. When too much current flows (a fault), the breaker trips. Power to that circuit stops. The rest of your house keeps running. Once the fault is fixed, you reset the breaker.

In software, the circuit breaker wraps calls to a downstream service. When that service starts failing, the breaker "trips" — it stops forwarding requests and returns errors immediately. Your service stays responsive, other features keep working, and the downstream service gets breathing room to recover.

Circuit Breaker States Deep Dive

A circuit breaker is a state machine with three states. Understanding the transitions is essential to tuning one correctly.

Closed ──[failures ≥ threshold]──► Open
  ▲                                  │
  │                             [timeout expires]
  │                                  │
  │                                  ▼
  └──[probe requests succeed]── Half-Open
                                     │
                         [probe requests fail]
                                     │
                                     ▼
                                   Open

Closed State (Normal Operation)

All requests pass through to the downstream service. The breaker counts failures silently in the background. A failure is any exception or timeout — you define what counts. Successes reset (or decay) the failure counter depending on your implementation.

Key parameters:

• failure_threshold — how many failures before opening (e.g., 5 in a 60-second window)
• failure_rate_threshold — alternative approach: open when failure rate exceeds X% (e.g., 50% of last 10 calls)

Open State (Failing Fast)

The breaker is tripped. No requests reach the downstream service at all. Every call immediately raises a CircuitOpenError (or similar). This is the critical behavior — fail in microseconds, not after a 30-second timeout.

The open state has a built-in timer. After the configured timeout (e.g., 30 seconds), the breaker automatically transitions to half-open to test whether the service has recovered. The timer starts from the last recorded failure, not from when the breaker opened.

Key parameters:

• timeout / reset_timeout — seconds to wait in open state before trying again (typical: 30–120 seconds)

Half-Open State (Probing Recovery)

The breaker lets a limited number of "probe" requests through. If those succeed, recovery is confirmed and the breaker closes. If they fail, the breaker opens again and the timeout restarts.

This state prevents a thundering herd: without half-open, the moment a service recovers, every queued request would slam it simultaneously. Half-open sends a controlled trickle first.

Key parameters:

• max_requests — how many probe requests to allow in half-open (typical: 1–5)
• success_threshold — consecutive successes needed to fully close (typical: 2–3)

Why This Matters: Cascading Failures

Without a circuit breaker, one slow service causes a cascade:

1. Payment service responds slowly (5s instead of 200ms)
2. Checkout service waits — thread blocked
3. More requests arrive — more threads blocked
4. Checkout service exhausts its thread pool and stops responding
5. API gateway waits for checkout — its threads block
6. Everything fails

With a circuit breaker:

1. Payment service starts failing
2. After 5 failures, circuit opens
3. All subsequent calls fail instantly (not after 5-second timeout)
4. Checkout service stays responsive — returns a useful error immediately
5. Other features continue working
6. After 30 seconds, breaker probes payment — if recovered, resets automatically

Implementing Circuit Breaker in Python

Here is a production-ready circuit breaker implementation with proper state machine, failure rate window, and thread safety:

import time
import threading
from enum import Enum
from collections import deque
from typing import Callable, Any, Optional
 
 
class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half-open"
 
 
class CircuitBreakerError(Exception):
    """Raised when the circuit is open and a call is attempted."""
    pass
 
 
class CircuitBreaker:
    """
    Thread-safe circuit breaker with sliding window failure tracking.
    
    Args:
        failure_threshold: Number of failures before opening the circuit
        timeout: Seconds to wait in OPEN state before probing (half-open)
        success_threshold: Consecutive successes in HALF_OPEN to close the circuit
        window_size: Number of recent calls to track for failure rate
    """
    def __init__(
        self,
        failure_threshold: int = 5,
        timeout: int = 60,
        success_threshold: int = 2,
        window_size: int = 10,
    ):
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.success_threshold = success_threshold
        self.window_size = window_size
 
        self._state = State.CLOSED
        self._failure_count = 0
        self._success_count = 0
        self._last_failure_time: Optional[float] = None
        self._call_window: deque = deque(maxlen=window_size)  # True=success, False=failure
        self._lock = threading.Lock()
 
    @property
    def state(self) -> State:
        return self._state
 
    def call(self, func: Callable, *args, **kwargs) -> Any:
        with self._lock:
            if self._state == State.OPEN:
                if self._should_attempt_reset():
                    self._transition_to_half_open()
                else:
                    raise CircuitBreakerError(
                        f"Circuit is OPEN — {func.__name__} unavailable. "
                        f"Retrying in {self._seconds_until_reset():.0f}s"
                    )
 
        try:
            result = func(*args, **kwargs)
            self._record_success()
            return result
        except Exception as exc:
            self._record_failure()
            raise
 
    def _should_attempt_reset(self) -> bool:
        if self._last_failure_time is None:
            return True
        return (time.monotonic() - self._last_failure_time) >= self.timeout
 
    def _seconds_until_reset(self) -> float:
        if self._last_failure_time is None:
            return 0
        elapsed = time.monotonic() - self._last_failure_time
        return max(0, self.timeout - elapsed)
 
    def _transition_to_half_open(self):
        self._state = State.HALF_OPEN
        self._success_count = 0
 
    def _record_success(self):
        with self._lock:
            self._call_window.append(True)
            if self._state == State.HALF_OPEN:
                self._success_count += 1
                if self._success_count >= self.success_threshold:
                    self._state = State.CLOSED
                    self._failure_count = 0
            elif self._state == State.CLOSED:
                self._failure_count = 0
 
    def _record_failure(self):
        with self._lock:
            self._call_window.append(False)
            self._failure_count += 1
            self._last_failure_time = time.monotonic()
            if self._failure_count >= self.failure_threshold:
                self._state = State.OPEN
 
    def get_stats(self) -> dict:
        """Return current breaker stats for monitoring."""
        with self._lock:
            window = list(self._call_window)
            total = len(window)
            failures = window.count(False)
            return {
                "state": self._state.value,
                "failure_count": self._failure_count,
                "success_count": self._success_count,
                "failure_rate": (failures / total * 100) if total > 0 else 0,
                "seconds_until_reset": self._seconds_until_reset(),
            }
 
 
# ── Usage ──────────────────────────────────────────────────────────────────
payment_breaker = CircuitBreaker(failure_threshold=5, timeout=30, success_threshold=2)
 
 
def process_payment(order_id: str, amount: float):
    try:
        return payment_breaker.call(payment_service.charge, order_id, amount)
    except CircuitBreakerError:
        # Circuit is open — fail fast with a user-friendly message
        raise PaymentUnavailableError("Payment service temporarily unavailable")
    except Exception:
        # Actual payment error — breaker already recorded the failure
        raise

Implementing Circuit Breaker in Node.js

Using the opossum library — the most widely used circuit breaker for Node.js:

const CircuitBreaker = require('opossum');
const axios = require('axios');
 
// The function you want to protect
async function callPaymentService(orderId, amount) {
  const response = await axios.post('https://payment-api/charge', {
    orderId,
    amount,
  });
  return response.data;
}
 
// Wrap it with a circuit breaker
const options = {
  timeout: 3000,              // 3 seconds before considering it a failure
  errorThresholdPercentage: 50, // Open when 50% of requests fail
  resetTimeout: 30000,        // 30 seconds before trying again (half-open)
  volumeThreshold: 5,         // Minimum calls before checking failure rate
};
 
const paymentBreaker = new CircuitBreaker(callPaymentService, options);
 
// Fallback: what to do when circuit is open
paymentBreaker.fallback((orderId, amount) => {
  console.warn(`Payment circuit open — queuing order ${orderId}`);
  return queueForRetry(orderId, amount);   // or return cached/default
});
 
// Event hooks for monitoring
paymentBreaker.on('open',     () => console.error('Payment circuit OPENED'));
paymentBreaker.on('close',    () => console.info('Payment circuit CLOSED'));
paymentBreaker.on('halfOpen', () => console.info('Payment circuit HALF-OPEN'));
paymentBreaker.on('fallback', (result) => console.warn('Fallback executed', result));
 
// Use it
async function processOrder(orderId, amount) {
  try {
    return await paymentBreaker.fire(orderId, amount);
  } catch (err) {
    if (paymentBreaker.opened) {
      throw new Error('Payment service temporarily unavailable');
    }
    throw err;
  }
}
 
// Health check endpoint
app.get('/health', (req, res) => {
  res.json({
    payment: {
      state: paymentBreaker.opened ? 'open' :
             paymentBreaker.halfOpen ? 'half-open' : 'closed',
      stats: paymentBreaker.stats,
    }
  });
});

Circuit Breaker Libraries

Do not implement circuit breakers from scratch in production unless you have a specific reason. Battle-tested libraries handle edge cases around thread safety, metrics, and configuration that custom implementations miss.

Quick rule of thumb: Use resilience4j for Java, Polly for .NET, opossum for Node.js, pybreaker for Python, gobreaker for Go. Use Istio/Envoy if you want zero-code circuit breaking across all services.

Circuit Breaker with Fallbacks

A circuit breaker without a fallback just converts one type of error into another. The real value comes from defining what to return when the circuit is open.

Option 1: Return Cached Response

Best for read operations where stale data is acceptable:

import functools
from datetime import datetime, timedelta
 
_cache: dict = {}
 
def get_user_profile(user_id: str) -> dict:
    cache_key = f"profile:{user_id}"
    try:
        result = profile_breaker.call(profile_service.get, user_id)
        # Update cache on success
        _cache[cache_key] = {"data": result, "cached_at": datetime.utcnow()}
        return result
    except CircuitBreakerError:
        # Return cached version if available and not too stale
        cached = _cache.get(cache_key)
        if cached and datetime.utcnow() - cached["cached_at"] < timedelta(hours=1):
            return {**cached["data"], "_from_cache": True}
        return {"error": "profile_unavailable", "user_id": user_id}

Option 2: Return Default/Degraded Response

Best when you can offer reduced functionality instead of an error:

def get_product_recommendations(user_id: str) -> list:
    try:
        return recommendation_breaker.call(recommendation_service.get, user_id)
    except CircuitBreakerError:
        # Return generic popular items instead of personalized ones
        return redis_client.get("global:popular_products") or DEFAULT_PRODUCTS
 
def get_shipping_estimate(cart: dict) -> dict:
    try:
        return shipping_breaker.call(shipping_service.estimate, cart)
    except CircuitBreakerError:
        # Show a range instead of an exact estimate
        return {"estimate": "3-7 business days", "exact": False}

Option 3: Queue for Later (Write Operations)

Best for non-time-critical writes where eventual consistency is acceptable:

def track_user_event(user_id: str, event: dict) -> None:
    try:
        analytics_breaker.call(analytics_service.track, user_id, event)
    except CircuitBreakerError:
        # Don't drop the event — queue it for when analytics recovers
        message_queue.publish("analytics.deferred", {
            "user_id": user_id,
            "event": event,
            "queued_at": time.time(),
        })

Netflix found that graceful degradation (showing less personalized content instead of an error) produced far better user experience than complete failure — users stayed engaged even when some features were degraded.

Circuit Breaker vs Retry Pattern

These are complementary patterns, not alternatives. Understanding when to use each — and how to combine them — is where most teams go wrong.

When to Use Retry

Retry is for transient errors — failures that resolve on their own within milliseconds to seconds:

• Brief network packet loss
• Momentary database lock contention
• Single-instance pod restart (load balancer hasn't updated yet)
• Rate limit hit (retry after delay)

import time
 
def retry_with_backoff(func, max_attempts=3, base_delay=0.5):
    for attempt in range(max_attempts):
        try:
            return func()
        except TransientError as e:
            if attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt)  # 0.5s, 1s, 2s
            time.sleep(delay)

When to Use Circuit Breaker

Circuit breaker is for sustained failures — when a service is clearly down and retrying makes things worse:

• Service has crashed and won't recover for 30+ seconds
• Database is overloaded and more traffic makes it worse
• Third-party API is having an outage
• Deployment gone wrong — entire service is returning 500s

Combining Them Correctly

The pattern is: retry first for transient errors, circuit breaker as the outer guard:

def call_with_resilience(func, *args, **kwargs):
    # Inner retry: handles transient errors (3 attempts, 500ms backoff)
    def retried():
        return retry_with_backoff(lambda: func(*args, **kwargs), max_attempts=3)
    
    # Outer circuit breaker: trips when retries keep failing
    return payment_breaker.call(retried)

Wrong combination: retry inside an open circuit breaker. If the circuit is open, do not retry — fail immediately. Check circuit state before retrying.

Right sequence: Circuit Breaker (outer) → Retry (inner) → Actual Call

Monitoring Circuit Breakers

A circuit breaker you cannot observe is a black box. You need metrics to know when breakers are tripping, how often, and for how long.

Essential Metrics to Track

# Prometheus metrics for a circuit breaker
from prometheus_client import Counter, Gauge, Histogram
 
# State gauge — 0=closed, 1=open, 2=half-open
circuit_state = Gauge(
    'circuit_breaker_state',
    'Current state of the circuit breaker',
    ['service_name']
)
 
# Call counters
circuit_calls_total = Counter(
    'circuit_breaker_calls_total',
    'Total calls through the circuit breaker',
    ['service_name', 'result']  # result: success | failure | rejected
)
 
# Failure rate gauge
circuit_failure_rate = Gauge(
    'circuit_breaker_failure_rate_percent',
    'Current failure rate in the sliding window',
    ['service_name']
)
 
# Time spent in open state
circuit_open_duration = Histogram(
    'circuit_breaker_open_duration_seconds',
    'How long the circuit stays open before recovering',
    ['service_name'],
    buckets=[5, 10, 30, 60, 120, 300, 600]
)
 
 
class InstrumentedCircuitBreaker(CircuitBreaker):
    def __init__(self, service_name: str, **kwargs):
        super().__init__(**kwargs)
        self.service_name = service_name
        self._open_since: Optional[float] = None
 
    def _record_success(self):
        super()._record_success()
        circuit_calls_total.labels(self.service_name, 'success').inc()
        self._update_state_metric()
 
    def _record_failure(self):
        was_closed = self._state == State.CLOSED
        super()._record_failure()
        circuit_calls_total.labels(self.service_name, 'failure').inc()
        if self._state == State.OPEN and was_closed:
            self._open_since = time.monotonic()
        self._update_state_metric()
 
    def _update_state_metric(self):
        state_value = {State.CLOSED: 0, State.OPEN: 1, State.HALF_OPEN: 2}
        circuit_state.labels(self.service_name).set(state_value[self._state])
        stats = self.get_stats()
        circuit_failure_rate.labels(self.service_name).set(stats['failure_rate'])

Alerting Thresholds

Grafana Dashboard Queries (PromQL)

# Is any circuit currently open?
circuit_breaker_state{} == 1
 
# Failure rate by service (last 5 minutes)
rate(circuit_breaker_calls_total{result="failure"}[5m])
/ rate(circuit_breaker_calls_total{}[5m]) * 100
 
# How many calls were rejected (circuit was open) per minute
rate(circuit_breaker_calls_total{result="rejected"}[1m])

Key Takeaways

• One slow service can cascade and take down your whole system
• Circuit breakers fail fast when a service is degraded, protecting everything else
• Three states: Closed (normal) → Open (failing fast) → Half-open (testing recovery)
• Half-open prevents thundering herd — it probes recovery with a small number of requests
• Always define a fallback: cached response, degraded response, or queue for retry
• Retry handles transient errors; circuit breaker handles sustained failures — use both, with circuit breaker as the outer wrapper
• Use a library (pybreaker, opossum, resilience4j, Polly) in production — don't reinvent this
• Instrument with Prometheus metrics: state, failure rate, rejected calls, open duration

Frequently Asked Questions

What is the circuit breaker pattern and why is it important?

The circuit breaker pattern is a fault-tolerance design pattern for distributed systems that prevents cascading failures by stopping traffic to a failing service. When a downstream service starts failing repeatedly, the circuit "trips" and all subsequent calls fail immediately (in microseconds) instead of waiting for a timeout (seconds). This keeps the calling service responsive, prevents thread pool exhaustion, and gives the failing service time to recover. Without circuit breakers, a single slow service can consume all available threads and crash healthy services that depend on it.

What are the three states of a circuit breaker?

A circuit breaker has three states: (1) Closed — normal operation, all requests pass through and failures are counted; (2) Open — the circuit has tripped, all requests fail immediately without reaching the downstream service; (3) Half-Open — after a timeout period, a small number of probe requests are allowed through to test whether the service has recovered. If probes succeed, the circuit closes. If probes fail, the circuit opens again and the timeout restarts.

How is a circuit breaker different from a retry?

Retry and circuit breaker solve different problems. Retry is for transient errors that resolve quickly (network hiccups, brief lock contention) — you retry a few times with exponential backoff. Circuit breaker is for sustained failures where the service is clearly down — retrying makes things worse by adding load. Use them together: wrap the function in retries for transient errors, then wrap that in a circuit breaker for sustained failures. If the circuit is open, skip retries entirely and fail fast.

What threshold should I set for opening the circuit?

Start with 5 consecutive failures or 50% failure rate over a 10-call sliding window, whichever comes first. Set the timeout (how long to stay open) at 30–60 seconds. Adjust based on your service's normal error rate — a service with 1–2% baseline errors needs a higher threshold than one that is normally error-free. Use time-based windows (failures per minute) rather than absolute counts for high-traffic services, to avoid circuits that never trip under low traffic.

How do I implement a circuit breaker in Node.js or Python?

For Node.js, use the opossum library: npm install opossum. Wrap your function with new CircuitBreaker(fn, options) and call it via breaker.fire(args). For Python, use pybreaker: pip install pybreaker. Decorate your function with @CircuitBreaker(fail_max=5, reset_timeout=60). Both libraries handle state management, timeouts, and metrics. Avoid writing your own implementation in production — the threading and state edge cases are subtle.

What is the half-open state in a circuit breaker?

Half-open is the recovery probe state. After the circuit has been open for the configured timeout period, it transitions to half-open and allows a small number of test requests through. If those requests succeed (above the success threshold), the circuit closes and normal traffic resumes. If they fail, the circuit opens again. Half-open prevents a thundering herd problem: without it, when a service recovers, every request that was queued up would hit it simultaneously, potentially overwhelming it and causing it to fail again immediately.

How do I monitor circuit breakers in production?

Track four key metrics: (1) circuit state (0=closed, 1=open, 2=half-open) as a gauge; (2) call outcomes (success, failure, rejected) as counters; (3) failure rate percentage over a sliding window; (4) time spent in open state per open event. Alert when a circuit opens (warning), stays open for more than 2 minutes (critical), or flaps open/closed more than 3 times in 10 minutes (critical — indicates a service that is recovering slowly). Most libraries like opossum and resilience4j expose Prometheus metrics out of the box.

Related reading: Rate Limiting · Load Balancing Strategies