Designing Idempotent APIs at Scale: The Hidden Backbone of Reliable Distributed Systems
Idempotency is one of the most overlooked yet critical properties in modern distributed systems. This post breaks down how to design truly idempotent APIs that survive retries, failures, and real-world chaos.
Introduction#
If you've ever built or operated a distributed system, you’ve likely faced this scenario: a request fails midway, the client retries, and suddenly you have duplicate records, double payments, or inconsistent state.
Welcome to the world of non-idempotent APIs.
In production systems—especially those involving payments, event processing, or microservices—idempotency is not optional. It’s a requirement for correctness.
Idempotency isn’t just about avoiding duplicates — it’s about making your system predictable under failure.
This post dives into practical patterns for designing idempotent APIs that actually work at scale.
What Does Idempotency Really Mean?#
An operation is idempotent if performing it multiple times produces the same result as performing it once.
In HTTP terms:
GET,PUT,DELETE→ should be idempotentPOST→ usually not idempotent
But in real-world systems, even POST endpoints (like payments or job submissions) must behave idempotently.
Where Things Break#
Let’s take a simple payment API:
POST /payments
{
"user_id": "123",
"amount": 100
}If the client times out and retries:
You might charge the user twice
Or create duplicate records
Or trigger downstream side effects multiple times
Now imagine this happening at scale under network instability. It gets ugly fast.
The Idempotency Key Pattern#
The most widely used solution is the Idempotency Key.
How it works:
The client generates a unique key per request
The server stores the result of the first request
Subsequent requests with the same key return the same response
Example Request:
POST /payments
Idempotency-Key: 7f3a9c2e-1234-4b2a-9f8d-abc123
{
"user_id": "123",
"amount": 100
}Implementation Strategy#
Step 1: Store the Key + Result
You need a storage layer (Redis, DB, etc.):
idempotency_key | request_hash | response | status |
|---|
Step 2: Validate Incoming Requests
If key exists:
Compare request payload (optional but recommended)
Return stored response
If not:
Process request
Store result
Example (Node.js + Express + Redis)#
Here’s a simplified implementation:
import express from "express";
import Redis from "ioredis";
import crypto from "crypto";
const app = express();
const redis = new Redis();
app.use(express.json());
function hashRequest(body) {
return crypto.createHash("sha256").update(JSON.stringify(body)).digest("hex");
}
app.post("/payments", async (req, res) => {
const key = req.headers["idempotency-key"];
if (!key) {
return res.status(400).json({ error: "Missing Idempotency-Key" });
}
const cached = await redis.get(key);
if (cached) {
const parsed = JSON.parse(cached);
// Optional: verify request consistency
if (parsed.requestHash !== hashRequest(req.body)) {
return res.status(409).json({ error: "Request payload mismatch" });
}
return res.status(200).json(parsed.response);
}
// Simulate processing (e.g., charging a card)
const result = {
paymentId: crypto.randomUUID(),
status: "success",
};
const payload = {
requestHash: hashRequest(req.body),
response: result,
};
await redis.set(key, JSON.stringify(payload), "EX", 60 * 60); // TTL 1 hour
res.status(201).json(result);
});
app.listen(3000, () => console.log("Server running"));Key Design Decisions#
1. TTL (Time-to-Live)
How long should you store idempotency keys?
Payments → 24–48 hours
Job submissions → shorter (1–2 hours)
Tradeoff:
Too short → duplicates slip through
Too long → storage bloat
2. Request Hashing
Why store a hash?
Because this is dangerous:
POST /payments
Idempotency-Key: abc
{ "amount": 100 }
POST /payments
Idempotency-Key: abc
{ "amount": 500 }Without validation, you’d return the wrong response.
3. Storage Choice
Redis → fast, ephemeral, ideal for high throughput
PostgreSQL → durable, transactional
DynamoDB → scalable, built-in TTL
Pick based on:
Throughput
Durability requirements
Cost constraints
Going Beyond APIs: Idempotency in Event-Driven Systems#
Idempotency is even more critical in event-driven architectures.
Think Kafka consumers:
Messages can be delivered more than once
Consumers must handle duplicates safely
Common Patterns:
Deduplication table
Idempotent writes (UPSERT)
Event versioning
Anti-Patterns to Avoid#
❌ “We’ll just trust the client not to retry”
This will fail in production.
❌ Using timestamps as idempotency keys
They’re not unique enough under concurrency.
❌ Ignoring downstream side effects
Even if your API is idempotent, your:
Email service
Payment gateway
Notification system
…might not be.
Observability Matters#
You need visibility into:
Duplicate request rates
Idempotency key collisions
Cache hit/miss ratio
Suggested metrics:
idempotency.cache.hitidempotency.cache.missidempotency.conflict
Real-World Insight#
At scale, retries are not edge cases—they’re the norm.
Between:
Network flakiness
Client retries
Load balancers
Timeouts
Your system will see duplicate requests.
Designing for idempotency means you're designing for reality.
Final Thoughts#
Idempotency is one of those concepts that seems simple but gets complex quickly when you consider edge cases, scale, and consistency.
If you're building:
Payment systems
Job queues
APIs consumed by mobile clients
…you should treat idempotency as a first-class design concern, not an afterthought.
Because in distributed systems, failure is guaranteed.
Predictability is not — unless you design for it.
4 comments