Stop Fearing Retries: Building Idempotent REST APIs

Have you ever stared at a bug report where a user was charged twice for the same order, or two identical user accounts were created milliseconds apart? If you build backend systems, you've probably been there.

The root cause is almost always the same: network instability combined with an impatient user (or an overly aggressive automated retry mechanism). When a client sends a request and doesn't get a response in time, it assumes the request failed and tries again. But what if the request didn't fail, and it was just the response that got lost?

Welcome to the world of distributed systems. Networks fail. Packets drop. Servers take slightly too long to respond. In this environment, retries are not just an edge case; they are an absolute necessity. However, if your API is not prepared to safely handle retries, you are sitting on a ticking time bomb.

This is where the concept of Idempotency comes to the rescue.

What Exactly is Idempotency?

In mathematics, an operation is idempotent if applying it multiple times produces the same result as applying it just once. Think of multiplying a number by 1, or taking the absolute value of a number.

In the context of REST APIs, an endpoint is idempotent if making multiple identical requests has the same effect on the server's state as making a single request.

By definition, some HTTP methods are inherently idempotent:

• GET: Retrieving data doesn't change state. You can call it 100 times, the database remains the same.
• PUT: Updating a resource with a specific payload. If you set status = 'active' once or ten times, the end state is always status = 'active'.
• DELETE: Deleting a record. The first call deletes it; subsequent calls might return a 404, but the end state (the record is gone) is exactly the same.

The real danger zone is POST. A POST request usually creates a new resource or triggers a state-changing action (like moving money). Calling it twice means creating two resources or moving money twice. We need a way to make POST requests idempotent.

The Solution: Idempotency Keys

To safely allow clients to retry POST requests, we need a mechanism to identify whether a request is genuinely new or just a retry of an earlier, perhaps delayed, request. We do this using an Idempotency Key.

Here is how the flow works in practice:

1. Client Generation: Before sending a critical request (like checkout), the client generates a unique identifier, usually a UUID v4.
2. Header Attachment: The client includes this identifier in the HTTP headers of the request, typically as Idempotency-Key: <uuid>.
3. Server Verification: When the server receives the request, it checks a database or cache (like Redis) to see if it has seen this exact key before.

Based on the check, the server handles three scenarios:

• Key Not Found (New Request): The server processes the request normally. Once completed, it stores the response payload and status code alongside the idempotency key, then returns the response.
• Key Found (Completed Request): The server sees the key and realizes it already processed this exact action. It bypasses the business logic entirely and simply returns the stored, original response.
• Key Found (In Progress): If a second request arrives while the first one is still processing, the server should recognize the lock and return an error (like 409 Conflict or 429 Too Many Requests), telling the client to back off and try later.

A Practical Example

Let's look at a simplified conceptual implementation using TypeScript and Express.

import express from 'express';
import { db, paymentGateway } from './services';

const app = express();
app.use(express.json());

app.post('/api/v1/payments', async (req, res) => {
  const idempotencyKey = req.header('Idempotency-Key');
  
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key header is required' });
  }

  // 1. Check if we have seen this request before
  const existingRecord = await db.getIdempotencyRecord(idempotencyKey);
  
  if (existingRecord) {
    // If it's already processing, prevent a race condition
    if (existingRecord.status === 'processing') {
      return res.status(409).json({ error: 'Request already in progress' });
    }
    
    // If it's done, return the cached response! No double charging.
    return res.status(existingRecord.statusCode).json(existingRecord.responseBody);
  }

  // 2. Mark as processing to prevent concurrent identical requests
  await db.createIdempotencyRecord({
    key: idempotencyKey,
    status: 'processing',
  });

  try {
    // 3. Perform the actual, dangerous business logic
    const paymentResult = await paymentGateway.charge(req.body.amount, req.body.source);
    
    // 4. Save the successful result
    await db.updateIdempotencyRecord(idempotencyKey, {
      status: 'completed',
      statusCode: 200,
      responseBody: paymentResult,
    });

    return res.status(200).json(paymentResult);
    
  } catch (error) {
    // On failure, delete the key so the user can retry, 
    // or store the failure state depending on business rules.
    await db.deleteIdempotencyRecord(idempotencyKey);
    return res.status(500).json({ error: 'Payment failed' });
  }
});

Real-World Traps to Avoid

While the concept is straightforward, the implementation in a production environment has a few gotchas.

1. The Database Transaction Trap

In our example above, there's a tiny window between processing the payment and saving the result to the database. If your server crashes right after the payment succeeds but before the database update, the user is charged, but your system thinks the key is still "processing" or missing.

To fix this, the business state change (e.g., updating the user's balance) and the idempotency record creation must be part of the same database transaction. If one fails, they both roll back. If you are communicating with external APIs (like Stripe), you often have to rely on their idempotency mechanisms combined with two-phase commit strategies or compensations.

2. Payload Mismatches

What if a client sends the same Idempotency-Key but changes the request body? (e.g., first request was for $50, the second is for $100). This is usually a bug on the client side. Your server should hash the incoming request body and compare it to the hash of the original request body. If they differ, immediately reject the request with a 400 Bad Request.

3. Don't Store Keys Forever

You don't need to keep idempotency records until the end of time. Most retries happen within minutes or hours of the original request. Storing keys in a fast key-value store like Redis with a Time-To-Live (TTL) of 24 to 48 hours is standard industry practice.

Conclusion

Building robust APIs isn't just about handling the happy path; it's about anticipating the countless ways things will fail. Network timeouts and client retries are guaranteed facts of life in software engineering.

By enforcing the use of idempotency keys for critical, state-changing endpoints, you shift retries from being a source of anxiety to a safe, expected part of your system's behavior. Your clients will thank you, your support team will thank you, and you will sleep much better at night knowing you aren't accidentally double-charging your users.

Start identifying the critical POST endpoints in your systems today, and make them idempotent. It's a small architectural change that yields massive dividends in system reliability.