A distributed lock with a time-to-live (TTL) lease prevents deadlocks when a process crashes, but it cannot guarantee mutual exclusion on its own. Whenever an unpredicted execution pause occurs—such as a stop-the-world garbage collection pause, CPU starvation, or network stall—the lock expires silently, allowing stale workers to overwrite fresh data.
To ensure data correctness in distributed systems, the lock manager must issue a monotonically increasing fencing token that downstream storage systems enforce on every write.
The Illusion of Safety in TTL-Based Locks
The standard recipe for distributed locking in backend services usually starts with a key-value store like Redis or Valkey. A worker acquires a lock with an expiration window:
SET lock:invoice:42 "worker-1" NX PX 10000
This command tells the system: grant the lock to worker-1 for 10 seconds if nobody else holds it. If worker-1 finishes in 200 milliseconds, it deletes the key, and everything works fine. If worker-1 crashes, the 10-second TTL expires automatically, freeing up the resource so other workers do not wait indefinitely.
The problem is that asynchronous distributed systems do not have synchronized physical clocks or predictable execution speeds. A lease expiration guarantees only that the lock service considers the lease abandoned; it does not freeze or terminate the worker holding it.
How Silent Data Corruption Occurs
Consider what happens during a routine heavy workload:
- Worker A acquires
lock:invoice:42with a 10-second lease. - Worker A begins reading an invoice, computing financial ledgers, and preparing database updates.
- Worker A encounters a 12-second garbage collection pause or hypervisor disk latency stall.
- Meanwhile, the 10-second TTL expires on Redis.
- Worker B attempts to acquire
lock:invoice:42, succeeds cleanly, computes a fresh ledger state, and commits writeV2to the database. - Worker A wakes up from the pause. Because local code execution resumes at the next instruction without inspecting the wall clock, Worker A believes it still holds the lock and writes its outdated ledger
V1to the database.
Worker A overwrote Worker B's valid state. Heartbeat background threads (often called "lock auto-renewers" or "watchdogs") alleviate brief delays, but they cannot prevent stalls that freeze the renewal thread alongside the main thread.
The Fencing Token Solution
In his analysis of distributed lock safety, Martin Kleppmann demonstrated that time-based guarantees cannot ensure mutual exclusion in asynchronous networks. The receiving storage engine must participate in fencing.
A fencing token is a strictly monotonically increasing counter issued alongside every granted lock:
- When Worker A acquires the lock, the lock manager assigns token
101. - When Worker A stalls and the lease expires, Worker B acquires the lock and receives token
102. - Worker B writes to the database with token
102. The database records that the latest applied token is102. - When Worker A wakes up and attempts to commit its write tagged with token
101, the database compares101 < 102and rejects the transaction immediately.
Lock Service Worker A (Token 101) Worker B (Token 102) Database (Last Token: 0)
| | | |
|--- Lock Granted (101)->| | |
| | [12s Pause...] | |
| (TTL Expires) | | |
|--- Lock Granted (102)----------------------------->| |
| | |--- Write Data (Token 102)-->| (Accept: 102 > 0)
| | | | [Store Token: 102]
| |--- Write Data (Token 101)------------------------------>| (Reject: 101 < 102)
Implementing Fencing Verification in Storage
Implementing fencing requires your target storage layer to enforce conditional writes based on the token. In SQL databases, this can be achieved with a dedicated fencing column:
UPDATE invoices
SET
status = 'PROCESSED',
total_amount = $amount,
last_fencing_token = $current_token
WHERE
id = $invoice_id
AND last_fencing_token < $current_token;
If the row count affected by this query is zero, the application knows a newer worker has already superseded its work. The transaction is aborted safely without corrupting rows.
In document databases or object stores (like S3 or DynamoDB), conditional expressions (such as DynamoDB's attribute_exists / fencing_token < :token or S3 If-Match ETags) serve the identical role.
async function updateWithFencing(invoiceId: string, data: InvoiceData, token: number) {
const result = await db.query(
`UPDATE invoices
SET status = $1, total = $2, last_fencing_token = $3
WHERE id = $4 AND last_fencing_token < $3`,
[data.status, data.total, token, invoiceId]
);
if (result.rowCount === 0) {
throw new Error(`Stale lock detected. Fencing token ${token} rejected.`);
}
}
Efficiency Locks vs. Correctness Locks
When designing distributed workflows, engineers must distinguish between two lock requirements:
- Efficiency Locks: Used to avoid redundant work (e.g., preventing two workers from rendering the same thumbnail or sending duplicate cache warm-up requests). For efficiency locks, an occasional concurrent execution is harmless. Standard Redis TTL locks without fencing tokens are sufficient and cheap.
- Correctness Locks: Used when concurrent writes cause irreversible corruption (e.g., billing, ledger settlement, inventory deductions, balance transfers). For correctness, relying solely on TTL locks is dangerous.
If your storage backend cannot validate incoming sequence tokens (such as third-party non-transactional webhooks or legacy APIs), distributed locks alone cannot make the operation safe against network pauses. For absolute data safety, enforce monotonic ordering where the data actually lives.
Loading comments…