Idempotency in APIs: Preventing Duplicate Operations

Idempotency in APIs means that making the same request multiple times produces the same result as making it once — no duplicate charges, no double-created orders, no corrupted state. It is the property that makes retries safe in distributed systems where networks fail and clients cannot tell whether their request was processed.

Your user clicks "Pay Now." The request goes out. Network hiccup — the response never arrives. Did the payment go through?

They click again. Now did they pay twice?

This is the idempotency problem — and every production API needs a solution for it.

What Is Idempotency?

An operation is idempotent if doing it multiple times has the same result as doing it once. The term comes from mathematics: applying a function repeatedly yields the same output as applying it once.

plaintext

Idempotent:
  Set price = $100         → Run 3 times → price = $100 ✓
  DELETE /orders/123       → Run 3 times → order deleted once ✓
  GET /users/123           → Run 100 times → same user data ✓
 
Not idempotent:
  Increment counter by 1   → Run 3 times → counter +3, not +1 ✗
  POST /orders (new order) → Run 3 times → 3 orders created ✗
  Charge card $100         → Run 3 times → $300 charged ✗

The critical insight: the observable effect must be the same, not the response itself. A DELETE returning 404 on the second call is still idempotent — the resource is gone either way.

HTTP Methods and Idempotency

HTTP specifies which methods should be idempotent. This is a contract between client and server — if your API violates it, you break clients that assume safe retry behavior.

Method	Idempotent	Safe (Read-Only)	Notes
GET	Yes	Yes	Pure read — no side effects
HEAD	Yes	Yes	Same as GET, no body
OPTIONS	Yes	Yes	Metadata only
PUT	Yes	No	Replace entire resource — same result every time
DELETE	Yes	No	Delete once or N times — resource is gone
POST	No	No	Creates new resources by default
PATCH	No	No	Partial updates — "increment by 1" is not idempotent

Why this matters for API design:

When a client (browser, mobile app, retry middleware) sees a network error, it checks the HTTP method before retrying. A GET is always safe to retry. A PUT is safe. A POST is not — which is why browsers show "Resubmit this form?" dialogs.

If you use POST for everything (a common anti-pattern), you lose this automatic retry-safety. Clients cannot retry blindly, so they either don't retry (causing lost operations) or retry unsafely (causing duplicates).

PATCH and idempotency: PATCH is technically not idempotent, but it can be if your patch format expresses absolute state. PATCH /user/123 {"email": "new@example.com"} is idempotent. PATCH /user/123 {"incrementPoints": 10} is not. Design your PATCH semantics carefully.

Why It Matters: Retries Are Inevitable

Networks fail. Servers crash. Clients timeout. In any distributed system, you need retries. But retries on non-idempotent operations cause duplicates.

The failure scenarios:

Request sent, server never received it → retry is safe
Server processed it, response lost in transit → retry causes duplicate
Server is slow, client times out and retries → server may process twice

You can't tell which scenario happened from the client's side. Idempotency lets you safely retry without checking first.

In microservices architectures this compounds: a single user action may trigger 5 downstream service calls, each with its own retry logic. Without idempotency at each hop, a single network blip can cause cascading duplicates across your entire system.

Idempotency Keys

The standard pattern for making non-idempotent operations safe: the client sends a unique key with the request. The server uses it to detect duplicates.

plaintext

Client → POST /payments
         Idempotency-Key: uuid-abc-123
         Body: { amount: 100, card: ... }
 
Server:
  1. Check if uuid-abc-123 was seen before
  2. If yes → return stored response (don't process again)
  3. If no → process payment, store response against uuid-abc-123

The key insight: the server stores the response from the first successful processing. On retry, it returns the stored response without re-executing the operation.

Key generation rules:

Unique per operation attempt: Two different payment attempts = two different keys
Stable across retries: Retry of the same attempt = same key
Unpredictable: Use UUID v4 or cryptographic random — never sequential integers

python

import uuid
import hashlib
 
# Recommended: UUID per user action (generate once, store client-side)
key = str(uuid.uuid4())
 
# Deterministic alternative: hash of stable operation parameters
# Only use if you can guarantee the parameters never change across retries
key = hashlib.sha256(
    f"{user_id}:{order_id}:{amount_cents}".encode()
).hexdigest()

Implementing Idempotency Keys

There are two main storage backends for idempotency records: Redis (fast, ephemeral) and a relational database (durable, queryable). Production systems often use both.

Redis-Backed Implementation (Python FastAPI)

python

import redis
import json
import uuid
from fastapi import FastAPI, Header, HTTPException, Request, Response
from functools import wraps
from datetime import timedelta
 
app = FastAPI()
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
 
IDEMPOTENCY_TTL = 86400  # 24 hours in seconds
 
def require_idempotency_key(func):
    @wraps(func)
    async def wrapper(*args, idempotency_key: str = Header(None), **kwargs):
        if not idempotency_key:
            raise HTTPException(
                status_code=400,
                detail="Idempotency-Key header is required for this endpoint"
            )
 
        cache_key = f"idem:{idempotency_key}"
        lock_key = f"idem_lock:{idempotency_key}"
 
        # Fast path: check cache before acquiring lock
        cached = r.get(cache_key)
        if cached:
            stored = json.loads(cached)
            return stored["body"]  # Return original response body
 
        # Acquire distributed lock to prevent concurrent processing
        with r.lock(lock_key, timeout=30, blocking_timeout=5):
            # Double-check inside lock (race condition guard)
            cached = r.get(cache_key)
            if cached:
                stored = json.loads(cached)
                return stored["body"]
 
            # Execute the actual handler
            result = await func(*args, idempotency_key=idempotency_key, **kwargs)
 
            # Store result with TTL
            r.setex(
                cache_key,
                IDEMPOTENCY_TTL,
                json.dumps({"body": result, "key": idempotency_key})
            )
            return result
 
    return wrapper
 
 
@app.post("/payments")
@require_idempotency_key
async def create_payment(
    request: Request,
    idempotency_key: str = Header(None)
):
    body = await request.json()
    payment_id = await process_payment(body["amount"], body["card_token"])
    return {"payment_id": payment_id, "status": "success"}

Database Deduplication Table (PostgreSQL + SQLAlchemy)

Redis is fast but volatile. For payments and financial operations, pair Redis with a database deduplication table that survives restarts:

python

from sqlalchemy import Column, String, JSON, DateTime, Index
from sqlalchemy.orm import Session
from datetime import datetime, timedelta, timezone
 
class IdempotencyRecord(Base):
    __tablename__ = "idempotency_records"
 
    key = Column(String(255), primary_key=True)
    endpoint = Column(String(255), nullable=False)
    response_body = Column(JSON, nullable=False)
    status_code = Column(Integer, default=200)
    created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
    expires_at = Column(DateTime(timezone=True), nullable=False)
 
    __table_args__ = (
        Index("idx_idempotency_expires_at", "expires_at"),  # For cleanup job
    )
 
 
def get_or_process(
    db: Session,
    key: str,
    endpoint: str,
    handler,
    ttl_hours: int = 24
):
    """
    Fetch cached response or process and store.
    Uses SELECT FOR UPDATE to prevent concurrent inserts.
    """
    now = datetime.now(timezone.utc)
 
    # Check for existing record (with row-level lock)
    record = (
        db.query(IdempotencyRecord)
        .filter(
            IdempotencyRecord.key == key,
            IdempotencyRecord.expires_at > now
        )
        .with_for_update()
        .first()
    )
 
    if record:
        return record.response_body, record.status_code
 
    # Process the operation
    result, status_code = handler()
 
    # Persist idempotency record
    record = IdempotencyRecord(
        key=key,
        endpoint=endpoint,
        response_body=result,
        status_code=status_code,
        expires_at=now + timedelta(hours=ttl_hours),
    )
    db.add(record)
    db.commit()
 
    return result, status_code

Node.js Express Middleware

javascript

const redis = require('redis');
const { v4: uuidv4 } = require('uuid');
 
const redisClient = redis.createClient({ url: process.env.REDIS_URL });
const IDEMPOTENCY_TTL = 86400; // 24 hours
 
/**
 * Express middleware for idempotency key enforcement.
 * Requires: Idempotency-Key header on POST/PATCH requests.
 */
function idempotencyMiddleware(options = {}) {
  const { ttl = IDEMPOTENCY_TTL, required = true } = options;
 
  return async (req, res, next) => {
    const idempotencyKey = req.headers['idempotency-key'];
 
    if (!idempotencyKey) {
      if (required) {
        return res.status(400).json({
          error: 'Idempotency-Key header is required'
        });
      }
      return next();
    }
 
    const cacheKey = `idem:${idempotencyKey}`;
 
    try {
      // Check for cached response
      const cached = await redisClient.get(cacheKey);
      if (cached) {
        const { statusCode, body } = JSON.parse(cached);
        return res.status(statusCode).json(body);
      }
 
      // Intercept the response to cache it
      const originalJson = res.json.bind(res);
      res.json = async (body) => {
        // Store response before sending
        await redisClient.setEx(
          cacheKey,
          ttl,
          JSON.stringify({ statusCode: res.statusCode, body })
        );
        return originalJson(body);
      };
 
      next();
    } catch (err) {
      // On Redis failure, allow request through (fail open)
      console.error('Idempotency cache error:', err);
      next();
    }
  };
}
 
// Usage
app.post('/payments', idempotencyMiddleware({ required: true }), async (req, res) => {
  const { amount, cardToken } = req.body;
  const payment = await processPayment(amount, cardToken);
  res.status(201).json({ paymentId: payment.id, status: 'success' });
});

Idempotency in Payment Systems

Payments are the canonical use case for idempotency. Money movement is irreversible — a duplicate charge can cause real financial and reputational harm.

How Stripe Implements Idempotency Keys

Stripe requires idempotency keys on all mutating API calls. Their implementation has several properties worth copying:

Every POST supports Idempotency-Key: The header is standard across all Stripe endpoints, not opt-in per route.

Keys are scoped to your API key: idem-key-123 from test mode and live mode are different records. Keys from different Stripe accounts never conflict.

24-hour retention window: Stripe stores idempotency records for 24 hours. After expiry, the same key would be treated as a new request.

Conflict detection: If you send the same idempotency key with different request bodies, Stripe returns a 422 error. This catches bugs where clients accidentally reuse keys for different operations.

python

import stripe
 
stripe.api_key = "sk_live_..."
 
def create_payment_intent(order_id: str, amount_cents: int, currency: str = "usd"):
    """
    Idempotency key tied to the specific order + attempt.
    Retrying for the same order reuses the same key → safe.
    """
    idempotency_key = f"pi_order_{order_id}"
 
    try:
        intent = stripe.PaymentIntent.create(
            amount=amount_cents,
            currency=currency,
            idempotency_key=idempotency_key
        )
        return intent
 
    except stripe.error.IdempotencyError as e:
        # You sent same key with different parameters — bug in your code
        raise ValueError(f"Idempotency key reused with different params: {e}")
 
    except stripe.error.APIConnectionError:
        # Network error — safe to retry with same idempotency_key
        raise  # Let the retry layer handle it
 
 
# First call: processes the payment
intent = create_payment_intent("order_456", 2999)
 
# Network timeout — user retries:
intent = create_payment_intent("order_456", 2999)  # Same result, no duplicate charge

Why Payments Specifically Require This

Payments involve external state (the payment network, the bank) that cannot be rolled back atomically. If your server processes a charge but crashes before writing the result to your database, you have:

Payment network: charge succeeded
Your database: no record of the charge

On retry, a naive system would charge again. With idempotency keys, the payment network (or your payment processor) recognizes the duplicate and returns the original result.

This "at-least-once delivery with idempotent processing" is the standard pattern for any operation touching external state — emails, SMS notifications, webhook deliveries, and financial transactions.

The Concurrent Request Problem

What if the same idempotency key arrives twice at the same time — two network retries hitting different server instances simultaneously?

Without coordination, both requests will:

Check the cache → both miss (nothing stored yet)
Both process the operation → duplicate!
Both write the result

You need a distributed lock:

python

import redis.exceptions
 
def process_idempotent(key: str, handler, ttl: int = 86400):
    cache_key = f"idem:{key}"
    lock_key = f"idem_lock:{key}"
 
    # Fast path: check cache before acquiring lock
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)
 
    try:
        # Acquire lock with timeout (prevents deadlocks)
        with r.lock(lock_key, timeout=30, blocking_timeout=5):
            # Re-check inside lock: another request may have processed while we waited
            cached = r.get(cache_key)
            if cached:
                return json.loads(cached)
 
            # Only one request reaches here
            result = handler()
            r.setex(cache_key, ttl, json.dumps(result))
            return result
 
    except redis.exceptions.LockNotOwnedError:
        # Lock expired before we finished — this is a timeout scenario
        raise HTTPException(status_code=503, detail="Processing timeout, please retry")
    except redis.exceptions.LockError:
        # Could not acquire lock within blocking_timeout
        raise HTTPException(status_code=409, detail="Concurrent request detected, please retry")

In a database-only approach, use INSERT ... ON CONFLICT DO NOTHING (PostgreSQL) or a unique constraint on the idempotency key column — the database handles the concurrency for you.

sql

-- PostgreSQL: atomic insert with conflict handling
INSERT INTO idempotency_records (key, endpoint, response_body, expires_at)
VALUES ($1, $2, $3, NOW() + INTERVAL '24 hours')
ON CONFLICT (key) DO NOTHING
RETURNING *;
 
-- If no rows returned, another process won the race — SELECT and return their result

Designing Idempotent APIs

Beyond the idempotency key pattern, design your endpoints to be naturally idempotent where possible:

Use PUT instead of POST for updates: PUT /users/123/email with {"email": "new@example.com"} is idempotent. Running it 3 times sets the same email. POST /users/123/update-email is ambiguous.

Avoid operations like "add 10 points": Instead, set the total — "set points to 150." The outcome is the same regardless of retries.

Return the same response on retry: Don't return 201 Created on the first call and 200 OK on retry. Some clients check status codes. Keep it consistent, or always return 200 with the resource state.

Use conditional requests for conflict detection: PUT /resources/123 with If-Match: "etag-value" ensures you're updating the version you think you are. This prevents lost updates in concurrent scenarios.

Testing Idempotent APIs

Idempotency is easy to claim and easy to break. Test it explicitly.

Unit Tests: Verify Same Response on Retry

python

import pytest
import uuid
 
@pytest.mark.asyncio
async def test_idempotent_payment_returns_same_result(client, mock_payment_processor):
    """Calling the same endpoint twice with the same key returns identical response."""
    key = str(uuid.uuid4())
    payload = {"amount": 1000, "card_token": "tok_test"}
 
    response_1 = await client.post(
        "/payments",
        json=payload,
        headers={"Idempotency-Key": key}
    )
    response_2 = await client.post(
        "/payments",
        json=payload,
        headers={"Idempotency-Key": key}
    )
 
    assert response_1.status_code == response_2.status_code
    assert response_1.json() == response_2.json()
    # Payment processor was called exactly once despite two requests
    assert mock_payment_processor.call_count == 1
 
 
@pytest.mark.asyncio
async def test_different_keys_create_different_operations(client):
    """Two requests with different keys are treated as separate operations."""
    payload = {"amount": 1000, "card_token": "tok_test"}
 
    response_1 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": str(uuid.uuid4())}
    )
    response_2 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": str(uuid.uuid4())}
    )
 
    # Both succeed, but with different payment IDs
    assert response_1.json()["payment_id"] != response_2.json()["payment_id"]
 
 
@pytest.mark.asyncio
async def test_missing_idempotency_key_rejected(client):
    """POST without Idempotency-Key header returns 400."""
    response = await client.post("/payments", json={"amount": 1000})
    assert response.status_code == 400
    assert "Idempotency-Key" in response.json()["detail"]

Testing Network Failure Scenarios

python

@pytest.mark.asyncio
async def test_idempotency_survives_processing_crash(client, redis_client):
    """
    Simulate: request processed, response stored, but client never got it.
    Verify: retry returns stored response without reprocessing.
    """
    key = str(uuid.uuid4())
    payload = {"amount": 500, "card_token": "tok_test"}
 
    # First request — succeeds
    response_1 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": key}
    )
    assert response_1.status_code == 201
 
    # Simulate client crashing before receiving response — retry with same key
    response_2 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": key}
    )
 
    assert response_2.status_code == 201
    assert response_1.json()["payment_id"] == response_2.json()["payment_id"]
 
 
@pytest.mark.asyncio
async def test_expired_idempotency_key_treated_as_new(client, redis_client):
    """After TTL expiry, same key is treated as a new request."""
    key = str(uuid.uuid4())
    payload = {"amount": 500, "card_token": "tok_test"}
 
    response_1 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": key}
    )
 
    # Manually expire the key in Redis
    await redis_client.delete(f"idem:{key}")
 
    response_2 = await client.post(
        "/payments", json=payload,
        headers={"Idempotency-Key": key}
    )
 
    # New payment created (key expired, treated as fresh request)
    assert response_1.json()["payment_id"] != response_2.json()["payment_id"]

Common Idempotency Mistakes

1. Using Timestamp as Idempotency Key

python

# WRONG: timestamp changes on every retry
key = f"payment_{datetime.now().isoformat()}"
 
# CORRECT: stable identifier generated once, persisted by client
key = str(uuid.uuid4())  # Generated before the first attempt, reused on retries

Timestamps are different on every retry by definition. The key must be generated before the first attempt and reused for all retries of that same operation.

2. Setting Too Short an Expiry

python

# WRONG: 60 seconds is too short for mobile retries, network outages
r.setex(cache_key, 60, json.dumps(result))
 
# CORRECT: 24 hours covers most retry windows
r.setex(cache_key, 86400, json.dumps(result))

If your TTL is shorter than your client's retry window, a retry after expiry will create a duplicate. Stripe uses 24 hours. For payments, consider 7 days — storage is cheap, duplicate charges are expensive.

3. Not Storing the Response — Only a "Processed" Flag

python

# WRONG: you know it was processed but can't return the original result
r.setex(f"idem:{key}", 86400, "processed")
# On retry: you know to skip, but you return what? A 200 with empty body?
 
# CORRECT: store the full response
r.setex(f"idem:{key}", 86400, json.dumps({
    "status_code": 201,
    "body": {"payment_id": "pay_123", "status": "success"}
}))
# On retry: return exact same status code and body

Clients check both status code and body. Returning a blank 200 instead of the original 201 with the payment ID breaks downstream processing.

4. Ignoring Concurrent Requests

A single Redis GET then SET with no locking creates a race condition window. Under load, two simultaneous retries will both miss the cache and both process. Always use a distributed lock or database-level unique constraint.

5. Scoping Keys Globally Instead of Per-User

python

# RISKY: if key generation has low entropy, users could collide
cache_key = f"idem:{idempotency_key}"
 
# BETTER: namespace by user/tenant
cache_key = f"idem:{user_id}:{idempotency_key}"

A user submitting Idempotency-Key: 1 should not collide with another user submitting Idempotency-Key: 1. Namespace your keys by authenticated user or tenant.

Key Takeaways

Idempotency: running an operation multiple times = same result as running once
GET, PUT, DELETE are idempotent by spec; POST and PATCH are not
Retries are unavoidable in distributed systems — idempotency makes them safe
Idempotency keys: client sends a unique ID; server detects and returns cached responses for duplicates
Store processed keys in Redis or database with a TTL (24 hours minimum, 7 days for payments)
Use distributed locks (Redis) or unique constraints (PostgreSQL) to prevent concurrent duplicates
Test explicitly: verify same key returns same response, and payment processor is called exactly once
Never use timestamps as idempotency keys — they change on every retry

Idempotency is what separates reliable APIs from ones that silently charge users twice.

FAQ

What is idempotency and why does it matter in API design?

Idempotency means that performing the same operation multiple times has the same effect as performing it once. It matters in API design because networks are unreliable — clients must retry failed requests, and without idempotency, retries cause duplicate operations like double charges, duplicate orders, or corrupted data. An idempotent API makes retries safe by design.

What is an idempotency key and how do I implement one?

An idempotency key is a unique identifier the client generates and sends with a request (typically as an Idempotency-Key HTTP header). The server stores the result of the first successful processing and returns that stored result for any subsequent request with the same key — without re-executing the operation. Implement it by: (1) generating a UUID client-side before the first attempt, (2) storing the key and response in Redis or a database with a 24-hour TTL, and (3) checking the store before processing any incoming request.

Which HTTP methods are idempotent?

GET, HEAD, OPTIONS, PUT, and DELETE are idempotent. GET and HEAD are also "safe" (read-only with no side effects). POST is not idempotent by default — each call may create a new resource. PATCH is technically not idempotent, though it can be designed to be if the operation expresses absolute state rather than relative changes.

How does Stripe implement idempotency?

Stripe accepts an Idempotency-Key header on all mutating API calls (POST requests). When you send the same key twice, Stripe returns the stored response from the first call without re-executing the payment. Stripe scopes keys to your API key (test vs. live are separate), retains keys for 24 hours, and returns a 422 error if you reuse a key with different request body parameters — catching client-side bugs.

How long should I store idempotency keys?

24 hours is the industry standard (used by Stripe and most payment processors). For payment operations, consider extending to 7 days to cover edge cases like delayed mobile retries, offline scenarios, and long-running batch jobs. The cost of storing a key (a few hundred bytes) is negligible compared to the cost of a duplicate charge. Clean up expired records with a scheduled background job using a database index on expires_at.

How do I handle idempotency with distributed systems?

In distributed systems, multiple server instances may receive retries simultaneously. Use a distributed lock (Redis SET NX PX or redlock) to ensure only one instance processes a given idempotency key at a time. The pattern is: check cache → acquire lock → re-check cache inside lock → process → store → release lock. Alternatively, use a database with a unique constraint on the idempotency key column and INSERT ... ON CONFLICT DO NOTHING — the database handles concurrency for you.

What is the difference between idempotent and safe HTTP methods?

"Safe" methods (GET, HEAD, OPTIONS) are read-only — they have no side effects on the server. "Idempotent" methods produce the same result when called multiple times, but may have side effects. DELETE is idempotent (deleting the same resource twice leaves it deleted) but not safe (it modifies server state). All safe methods are idempotent, but not all idempotent methods are safe.

Idempotency in APIs: Preventing Duplicate Operations

What Is Idempotency?

HTTP Methods and Idempotency

Why It Matters: Retries Are Inevitable

Idempotency Keys

Implementing Idempotency Keys

Redis-Backed Implementation (Python FastAPI)

Database Deduplication Table (PostgreSQL + SQLAlchemy)

Node.js Express Middleware

Idempotency in Payment Systems

How Stripe Implements Idempotency Keys

Why Payments Specifically Require This

The Concurrent Request Problem

Designing Idempotent APIs

Testing Idempotent APIs

Unit Tests: Verify Same Response on Retry

Testing Network Failure Scenarios

Common Idempotency Mistakes

1. Using Timestamp as Idempotency Key

2. Setting Too Short an Expiry

3. Not Storing the Response — Only a "Processed" Flag

4. Ignoring Concurrent Requests

5. Scoping Keys Globally Instead of Per-User

Key Takeaways

FAQ

Enjoyed this article?

Related Posts

ACID Properties Explained: Database Transactions

Database Replication Explained: Primary, Replica, and Failover

Database Sharding Explained: Scale to Millions of Users