Designing Idempotent APIs Without Race Conditions

When a network request times out, a client faces a classic distributed systems dilemma: did the server fail before processing the mutation, or did the response get lost on the wire? Retrying a non-idempotent POST or PATCH without protection risks creating duplicate orders, charging a customer twice, or corrupting state.

While the IETF Idempotency-Key specification formalizes standard header conventions, implementing idempotency correctly requires much more than slapping a Redis cache in front of your controller. A naive implementation introduces severe race conditions and data corruption bugs.

The Pitfalls of Naive Caching

A common beginner mistake is treating the idempotency key as a simple lookup key for cached responses:

1. Check if idempotency_key exists in Redis.
2. If present, return the cached JSON response.
3. If absent, execute the business logic, cache the result, and respond.

This pattern breaks immediately under concurrent retries. If a client triggers two identical payment requests simultaneously—or if a mobile client disconnects and instantly retries while the first worker is still writing to the database—both workers perform step 1 at the same moment. Both see a cache miss, both proceed with the mutation, and the customer is charged twice.

An idempotency mechanism must act as an atomic reservation and synchronization lock, not just a passive read-through cache.

The Three-State Idempotency Lifecycle

To handle in-flight requests and partial failures safely, every idempotency record must move through a clear state machine:

• STARTED (or IN_PROGRESS): The key is claimed atomically by a worker. Any incoming request with the same key while in this state is recognized as a concurrent execution.
• RESOLVED: The business transaction committed successfully. The stored status code, headers, and response body can be safely replayed to identical retries.
• FAILED: An unrecoverable infrastructure failure occurred before business completion. The key can be released or marked for safe client retry.

When a second request arrives while the first is still STARTED, the server should either wait briefly on a lock or return an immediate 409 Conflict (or 429 Too Many Requests), instructing the client that processing is actively underway.

Request Fingerprinting

What happens if an attacker or a buggy client sends the same Idempotency-Key for two completely different payloads—say, paying $10 and then paying $1000?

If your system blindly matches on the key string alone, the second request might receive the cached response of the first, masking a severe logic error. To prevent this, every idempotency record must store a request fingerprint: a cryptographic hash (such as SHA-256) of the HTTP method, request path, authenticated user identity, and normalized request body.

Fingerprint = SHA256(Method + "|" + Path + "|" + UserID + "|" + RequestBody)

When a matching key is found in storage, compare its stored fingerprint against the incoming request. If the keys match but the fingerprints differ, return 422 Unprocessable Entity or 400 Bad Request with an explicit error indicating payload mismatch.

Atomic Storage in PostgreSQL

For transactional systems, storing idempotency records directly inside your primary relational database guarantees transactional atomicity without distributed lock coordination.

CREATE TABLE idempotency_records (
    key VARCHAR(255) NOT NULL,
    user_id UUID NOT NULL,
    request_hash CHAR(64) NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'STARTED',
    response_code INT,
    response_body JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    locked_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (user_id, key)
);

When an incoming mutation arrives, attempt an atomic reservation in the same database:

INSERT INTO idempotency_records (key, user_id, request_hash, status)
VALUES ($1, $2, $3, 'STARTED')
ON CONFLICT (user_id, key) DO UPDATE
SET locked_at = NOW()
WHERE idempotency_records.status = 'FAILED'
RETURNING status, request_hash, response_code, response_body;

If the insert succeeds, your worker owns the execution lock and proceeds with the core business logic. If the insert encounters an existing row:

1. Verify the request_hash. Return 422 if mismatched.
2. If status is RESOLVED, immediately replay the stored response_code and response_body.
3. If status is STARTED, return 409 Conflict with a Retry-After header.

Once your business transaction finishes, update the record within the same transaction:

UPDATE idempotency_records
SET status = 'RESOLVED',
    response_code = 201,
    response_body = $1
WHERE user_id = $2 AND key = $3;

Production Hardening

In real-world deployments, pay attention to these edge cases:

• Worker Crashes: If a worker crashes mid-execution while the status is STARTED, subsequent retries will be blocked. Add an expiration threshold (e.g., allow overriding keys stuck in STARTED for longer than 60 seconds).
• Retention and Cleanup: Idempotency keys should not live forever. Maintain a TTL—Stripe typically retains keys for 24 hours. Use a daily partition or scheduled deletion query to purge records older than a week.
• Scope Isolation: Always scope idempotency keys to an authenticated user_id or api_client_id in composite primary keys to prevent cross-tenant key pollution.

By treating idempotency as an atomic state machine rather than an afterthought cache, your APIs can safely endure erratic mobile networks, client retries, and gateway timeouts without corrupting user data.

Sources

1. vertexaisearch.cloud.google.com
2. vertexaisearch.cloud.google.com
3. vertexaisearch.cloud.google.com
4. datatracker.ietf.org
5. stripe.com