Stop Ignoring Idempotency: Handle Duplicate API Requests

Building resilient APIs means handling duplicate requests without causing data corruption. In this post, we will explore how to implement Idempotency Keys to prevent race conditions and double-charging.

The Chaos of Duplicate Requests

Picture this: Your user clicks the "Submit Order" button, but their 3G connection is struggling. The UI spins, freezes, and out of frustration, they click the button again. Bam! You have just charged their credit card twice. Customer support gets angry emails, the finance team has to manually issue refunds, and your engineering team looks bad.

This scenario happens all the time in distributed systems. Networks are flaky. Requests timeout. When a client encounters a timeout, it has no way of knowing if the server processed the request before the connection dropped, or if the request never reached the server at all.

The natural reaction for any robust client is to retry. But if the server already processed the first request, retrying will duplicate the action. How do we build APIs that safely handle these retries without causing data corruption or double-charging?

The answer lies in a concept called Idempotency.

What is Idempotency?

In mathematics, an idempotent operation is one that produces the same result regardless of how many times it is applied. For example, multiplying a number by 1, or taking the absolute value of a number.

In API design, an idempotent endpoint guarantees that making multiple identical requests will have the exact same effect on the system as making a single request.

By definition, HTTP methods like GET, PUT, and DELETE are naturally idempotent. If you send a DELETE /users/123 request ten times, the end state is always the same: the user is deleted.

However, POST requests are generally not idempotent. Every time you call POST /orders, the server assumes you want to create a brand new order. To make a POST endpoint idempotent, we need to teach the server how to recognize a retry. The server needs to think: "Wait, I have seen this exact request before. I will not process it again."

The Idempotency Key Pattern

The industry standard solution, heavily popularized by companies like Stripe, is the Idempotency Key pattern.

It works by introducing a custom HTTP header, usually named Idempotency-Key. Here is the step-by-step flow:

1. Client Generation: The client generates a unique identifier (like a UUIDv7) for a specific user action.
2. First Request: The client sends the API request, including the header Idempotency-Key: <uuid>.
3. Server Check: Before executing any business logic, the server checks a fast storage layer (like Redis or a Postgres table) to see if this key has been processed.
4. If Key Exists: The server skips the business logic. Instead, it retrieves the saved response from the previous successful execution and returns it to the client immediately.
5. If Key Does Not Exist: The server locks the key, executes the payment or order logic, saves the final response payload into the cache alongside the key, and returns the response.

Implementation Example in Node.js

Let's look at a practical pseudo-code example using Node.js, Express, and Redis to implement this logic.

async function processCheckout(req, res) {
  const idempotencyKey = req.header('Idempotency-Key');
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key header is required' });
  }

  // 1. Check if we already processed this request
  const cachedResponse = await redis.get(`idemp:${idempotencyKey}`);
  if (cachedResponse) {
    // Return the old successful response as if it just happened
    return res.status(200).json(JSON.parse(cachedResponse)); 
  }

  // 2. Prevent race conditions with an atomic lock
  const lock = await redis.set(`lock:${idempotencyKey}`, '1', 'NX', 'EX', 10);
  if (!lock) {
    // Another request with this key is currently executing
    return res.status(409).json({ error: 'Concurrent request is processing.' });
  }

  try {
    // 3. Execute the heavy/sensitive business logic
    const orderResult = await executePayment(req.body);
    
    // 4. Cache the result for future retries (e.g., store for 24 hours)
    await redis.set(`idemp:${idempotencyKey}`, JSON.stringify(orderResult), 'EX', 86400); 
    
    return res.status(201).json(orderResult);
  } finally {
    // 5. Release the lock when done
    await redis.del(`lock:${idempotencyKey}`);
  }
}

This middleware logic acts as a shield, protecting your core business functions from a whole class of nasty bugs.

Common Pitfalls to Avoid

While the pattern seems straightforward, developers often make a few critical mistakes during implementation.

1. Generating the Key on the Server

This is the most common mistake. The Idempotency Key must be generated by the Client (Frontend web app or Mobile app). If the server generates the key, a network timeout will force the client to retry without a key, causing the server to generate a brand new key for the second request. The server will treat it as two separate requests. The client should generate the UUID when the checkout screen mounts, and reuse it for all retries of that specific transaction.

2. Returning an Error on Retry

Assume the first request succeeds on the server, but the network drops before the client receives the response. The client retries. If your server sees the duplicate key and returns an HTTP 400 error like "Duplicate Request", the client will think the transaction failed! The server must store the original successful JSON response and return it directly, making the retry entirely transparent to the client.

3. Ignoring Race Conditions

If a user double-clicks rapidly, two identical requests might hit your server at the exact same millisecond. If your code simply does if (!redis.get(key)), both requests might see a null value and proceed to charge the user simultaneously. You must use an atomic operation—like Redis SET NX (Set if Not eXists) or a database row lock—to ensure only a single thread can enter the execution block.

The Last Line of Defense: Database Constraints

While Redis and application-level locks are fantastic for performance, you should never trust a single layer of defense. What happens if your Redis cluster crashes, restarts, or evicts your idempotency keys prematurely due to memory limits?

Your database must be the ultimate source of truth. Always combine idempotency keys with database-level constraints.

For example, in your SQL database, you can create a UNIQUE index on the idempotency_key column within your payments table. If the application layer somehow fails and lets a duplicate request through, the database will aggressively reject the insert with a Unique Constraint Violation. It is much better to throw an ugly 500 error than to silently double-charge a customer.

Stop Ignoring the Chaos

Idempotency is not a "nice-to-have" feature; it is a fundamental requirement for reliable distributed systems. Building resilient APIs means accepting that networks will fail, and clients will retry.

The next time you sit down to write a POST endpoint that mutates critical state—whether it is processing a payment, reserving inventory, or sending a welcome email—ask yourself: "What happens if this endpoint is called twice at the exact same millisecond?"

Stop ignoring idempotency. Adopt Idempotency Keys, rely on database constraints, and let your users click that submit button as frantically as they want.