You're on a train. You tap Place order, the spinner turns, the train enters a tunnel, and a while later the app says the request timed out. There's a Try again button.
Should that button be there?
It depends on something the app can't see. A timeout means the client stopped waiting. It says nothing about what the server did. Behind that one error are three different stories:
- The request never reached the server. Nothing happened. Retrying is right.
- It reached the server and failed. Nothing was saved. Retrying is right.
- It reached the server, the order was saved, and the response got lost on the way back. Retrying creates a second order.
From the client's side, all three look identical. Retry blindly and story 3 charges the customer twice. Never retry and story 1 loses the order. The only way out is to change the server so that retrying is always the right call.
Here's how a platform of thirteen NestJS services does that in the repository: idempotency keys for the same request sent twice, version checks for two people editing one record, and the case that needs both. I'll also be clear about where the implementation is still partial.
Was the order created? What a timeout hides
Think of a boarding gate. Your boarding pass has a barcode. The first scan lets you onto the plane. Scan it again, maybe because the agent didn't hear the beep, and the scanner doesn't give you a second seat. It says already boarded.
That's the property we want. The scanner can be relaxed about double scans because it keeps a log: barcode, outcome, time. The second scan is a lookup, not a second boarding.
An API gets the same property from two pieces:
- the client sends a key that names the operation, and sends the same key on every retry (the barcode);
- the server keeps a receipt for that key with the answer it gave, and replays that answer to any retry (the gate's log).
That's the whole idea. The rest is detail, and the bugs live in the detail.
What is an idempotency key, and who creates it?
An operation is idempotent if doing it twice leaves the system in the same state as doing it once. PUT /items/7 {"qty": 3} is naturally idempotent: setting a quantity to 3 twice still leaves 3. POST /orders isn't: every call creates a new order.
An idempotency key makes a non-idempotent call safe to repeat. The client sends it in a header, conventionally Idempotency-Key, which the IETF is standardising in the Idempotency-Key header draft.
The key must identify the user's intent, not the HTTP attempt. Getting this wrong is the most common mistake, and it takes one line:
// Wrong: every attempt gets a fresh key, so the server sees three unrelated requests
await retry(() =>
fetch('/api/v1/orders', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Idempotency-Key': crypto.randomUUID() },
body: JSON.stringify(order),
}),
);// Right: mint the key once, when the user taps the button, and reuse it on every retry
const idempotencyKey = crypto.randomUUID();
await retry(() =>
fetch('/api/v1/orders', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Idempotency-Key': idempotencyKey },
body: JSON.stringify(order),
}),
);If the app can be killed between attempts, store the key with the draft. A boarding pass reprinted with a new barcode every time you walk up to the gate isn't a boarding pass.
In the repository, the order service requires this header on order writes and accepts 16 to 100 printable ASCII characters. A UUID fits. 1 doesn't.
What makes two requests "the same"?
The key alone isn't enough. Three questions decide what the server does.
Whose key is it? Clients choose keys, so two clients can pick the same one. A receipt is scoped by caller, tenant and operation as well as the key, so two users who both send abc… never see each other's receipts.
Is it really the same request? A buggy client might reuse a key for a different body. The server stores a fingerprint of the first request (a hash of the method, the normalised path and the body) and compares it on every retry.
What happened last time? Still running, finished, or never started.
Together they give four cases:
| Key | Fingerprint | Earlier attempt | Server answers |
|---|---|---|---|
| New | — | None | Run it, save a receipt |
| Seen | Same | Finished | Replay the stored response |
| Seen | Same | Still running | "In progress, retry shortly" |
| Seen | Different | Any | Reject: the key was reused |
The fingerprint hides a trap. {"qty":2,"item":"A"} and {"item":"A","qty":2} are the same request with different bytes. Hash a canonical form with sorted keys, not the raw body:
import { createHash } from 'node:crypto';
type Json = null | boolean | number | string | Json[] | { [key: string]: Json };
function canonical(value: Json): string {
if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
if (value !== null && typeof value === 'object') {
const keys = Object.keys(value).sort();
return `{${keys.map((key) => `${JSON.stringify(key)}:${canonical(value[key])}`).join(',')}}`;
}
return JSON.stringify(value);
}
function fingerprint(method: string, path: string, body: Json): string {
return createHash('sha256').update(canonical({ method, path, body })).digest('hex');
}
console.log(
fingerprint('POST', '/orders', { qty: 2, item: 'A' }) ===
fingerprint('POST', '/orders', { item: 'A', qty: 2 }),
); // trueThe shared library in the repository does the same and also rejects non-finite numbers and cyclic objects.
Two smaller choices. The raw key never touches the table: the order service stores an HMAC-SHA256 of it, which means nothing without the server's secret. And the stored response is encrypted, because it carries the same personal data as the order and sits in the table for a day.
Where should the receipt live? Three designs, three failure windows
The real question isn't "do we store a receipt?" but "when does the receipt become true, relative to the work it describes?"
Design 1: check, act, remember
const existing = await db.receipt.findUnique({ where: { key } });
if (existing) return existing.response;
const order = await db.order.create({ data: input }); // two retries can both reach this line
await db.receipt.create({ data: { key, response: toDto(order) } });
return toDto(order);Retries don't wait politely for each other. A mobile client with an aggressive timeout fires attempt two while attempt one is still running. Both read "no receipt", and both create an order. The window between the check and the insert is wide open under load.
Design 2: claim first, complete later
Let a unique constraint pick a winner before any work starts:
// Only one attempt can insert the claim; the others hit the unique constraint
await db.receipt.create({ data: { key, status: 'IN_PROGRESS', leaseUntil: secondsFromNow(30) } });
const order = await db.order.create({ data: input }); // commits on its own
// <- the process dies here
await db.receipt.update({ where: { key }, data: { status: 'SUCCEEDED', response: toDto(order) } });Concurrent duplicates are solved, but the order and the receipt still commit separately. If the process dies between them, the order exists and the receipt says "in progress" until its lease expires. Then the next retry claims it and creates a second order. Releasing the claim on any error, after something durable already happened, opens the same hole.
This isn't hypothetical. An older, generic helper in the same repository has exactly this shape, and the decision record for this area names a fail-open gap found in static review. The record itself refuses to promise a platform-wide guarantee.
Design 3: commit the receipt with the first durable effect
Stop treating the receipt as a note about the work and make it part of the work. The order service's command receipts do this in two steps.
Claim in a short transaction: insert the receipt as IN_PROGRESS with a new claim token and a 30-second lease. If a receipt already exists, replay it, report "in progress", or take over an expired lease.
Execute in one transaction that contains everything durable:
return db.$transaction(
async (tx) => {
// Every attempt that touches this key waits here until the transaction ends
await tx.$executeRaw`SELECT pg_advisory_xact_lock(hashtextextended(${scope}, 0))`;
const receipt = await tx.receipt.findUniqueOrThrow({ where: { scope } });
if (receipt.status === 'SUCCEEDED') return receipt.response; // another attempt finished
if (receipt.claimToken !== claimToken) throw new ClaimLostError(); // our lease was taken over
const order = await tx.order.create({ data: input });
await tx.outboxEvent.create({ data: orderCreatedEvent(order) });
const response = { status: 201, body: toDto(order) };
await tx.receipt.update({ where: { scope }, data: { status: 'SUCCEEDED', response } });
return response;
},
{ isolationLevel: 'Serializable' },
);Now walk through the crashes:
- Crash before commit. Postgres rolls back the order, the event and the receipt update together. The receipt stays
IN_PROGRESSuntil its lease expires, then a retry takes over and does the work for the first time. - Crash after commit. The order and a
SUCCEEDEDreceipt exist together. The retry replays the stored 201. - Slow original, eager retry. The original stalls before its transaction, its lease expires, and a retry claims the receipt with a new token. When the original finally runs, the token check fails and it aborts. The token is a fence: only the current lease holder can write.
There's no state where the order exists without its receipt, or the other way round. The outbox row rides in the same transaction, so the event announcing the order can't be lost either. That's the transactional outbox.
One limit: this only works when the first durable effect lives in the same database as the receipt. A call to a payment provider can't be wrapped. Pass a key downstream instead, derived deterministically from yours so every retry sends the same one (derived UUIDv7s do this), or move the call behind the outbox and let a saga own it.
How do you prevent lost updates? Versions and If-Match
Idempotency keys handle one client saying the same thing twice. They do nothing about two clients saying different things about the same record.
Two operators open the same order. Both see the delivery note "leave at the door", version 1. Alice changes it to "call on arrival" and saves. Bob, still looking at version 1, fixes the contact phone and saves the whole form, old note included. Alice's change disappears, and nobody gets an error. That's a lost update.
The fix: every write states which version it's based on, and the database refuses a write based on an old one. You can run this in psql:
CREATE TABLE orders (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
note text NOT NULL,
version integer NOT NULL DEFAULT 1
);
INSERT INTO orders (note) VALUES ('leave at the door');
-- Alice and Bob both read version 1. Alice writes first.
UPDATE orders SET note = 'call on arrival', version = version + 1
WHERE id = 1 AND version = 1; -- UPDATE 1
-- Bob is still holding version 1.
UPDATE orders SET note = 'leave at the door', version = version + 1
WHERE id = 1 AND version = 1; -- UPDATE 0: nothing overwritten, report a conflictThe check and the write are one statement, so nothing can slip in between. The tempting alternative, reading the row, comparing versions in TypeScript and then updating, reopens exactly the gap the WHERE clause closes.
Over HTTP, the version travels as an entity tag. GET returns ETag: "1", the client sends PATCH with If-Match: "1", and the server puts that version in the WHERE. In the order service, a missing or malformed If-Match gets 428 Precondition Required: the server refuses to guess what the client was looking at.
Back at the gate: an agent moving you to seat 14C works from a seat map printed earlier. If the map has changed since, the move is rejected and they print a fresh one. That's optimistic locking. Nobody holds a lock while thinking; the rare conflict is caught at write time.
SELECT … FOR UPDATE, the pessimistic alternative, fits when the read and the write sit inside one short server-side transaction. It can't span the minute a person spends editing a form, which is exactly when lost updates happen.
Why do you need both? The retry that conflicts with its own ghost
This is the case that made me stop treating these as two separate features.
The client sends PATCH /orders/{id} with If-Match: "3". The server applies it, and the order becomes version 4. The response is lost in a tunnel. The client retries the identical request, still with If-Match: "3".
With version checks alone, the retry fails with a conflict. The client is in conflict with itself: the "other writer" is its own first attempt. A careful client that resolves conflicts by re-reading and re-applying its change could apply a relative change, such as "add one item", twice.
With a receipt in front, the retry never reaches the version check. Same key, same fingerprint, receipt SUCCEEDED: the server replays the original 200 together with ETag: "4". That works because the order service puts expectedVersion into the fingerprint and stores the ETag header with the response.
| Threat | Idempotency key | Version check |
|---|---|---|
| Same request arrives twice | Replays the first answer | Reports a false conflict |
| Two people edit one record | Can't see it | Rejects the stale write |
| Retry after a lost response | Replays with the new ETag | Needs the key in front |
So check the receipt first, then the version. The receipt answers "have I seen this exact request?" and the version answers "is this request based on current data?" Only a new request needs the second question.
Does the consumer side need the same thing?
Yes, because events arrive at least once. A consumer needs its own gate log: an inbox table with a unique event id, written in the same transaction as the effect, plus a per-aggregate version so a late, stale event can't overwrite newer state. The outbox article walks through those failure cases.
Under both sides sits one last line. The decision record keeps "a database constraint or transaction as the final invariant guard". Keys and versions make retries cheap; the constraint makes a bug loud.
How long should a receipt live?
Forever is expensive; too short brings the duplicates back. The order service keeps command receipts for 24 hours. When one expires, a cleanup job doesn't delete it. It erases the stored response and marks the row as purged, leaving a tombstone. The key's hash and the fingerprint stay, so a client retrying a day-old key gets "receipt expired" instead of a second order.
That's the airline again: after the flight, the gate log forgets the passenger details but still knows the barcode was scanned.
The older generic receipts in the same service simply delete expired rows and use a 5-minute lease instead of 30 seconds. That inconsistency is why the decision record says not to promise 24-hour retention across the platform.
Which status codes would I change?
Today every idempotency conflict in the order service is a 409 Conflict with its own error code, and a stale version is a 409 too. Clients can tell the cases apart by error code, but the HTTP status says less than it could:
| Situation | In the repository today | What I'd return |
|---|---|---|
| Key reused, different body | 409 IDEMPOTENCY_KEY_REUSED |
422, as the IETF draft suggests |
| Same key, still running | 409, retryAfterMs in the body |
409 with a Retry-After header |
Stale If-Match |
409 VERSION_CONFLICT |
412 Precondition Failed |
Missing If-Match |
428 | 428, unchanged |
The 412 change matters most. RFC 9110 defines what a failed If-Match returns, and generic HTTP clients already understand it. A 409 forces every client to special-case your API.
A quieter mismatch: one service expects If-Match: "7", another parses W/7. RFC 9110 evaluates If-Match with strong comparison, under which a weak tag never matches. I'd use strong tags everywhere, generated by one helper.
What it costs
- One more write per mutation, plus a lookup on every retry.
- A serialised path per key. The advisory lock is cheap, but the create transaction runs at
Serializableand is retried once on a write conflict. Normal traffic rarely hits the same key twice at once; a hot key would feel it. - Secrets to manage. Hashed keys and encrypted responses make key rotation part of the feature.
- Client discipline. None of this helps a client that mints a new key per attempt.
- Decisions nobody makes for you. Which failures get stored and what an expired key returns are product choices, not library defaults.
What I'd do differently
One implementation, not two generations. The order service has the generic helper (5-minute lease, plain JSON response, deleted on expiry, completed outside the business transaction) next to the newer command receipts (30-second lease, encrypted response, tombstones, completed inside it). Other services have their own variants. I'd put design 3 in the shared library before a second endpoint needed it.
Store only failures that would repeat. The command receipts record the failure they returned, so a retry with the same key replays the same error. That's right for "item unavailable". But when a transient failure rolled its transaction back, nothing happened, and I'd release the claim so the same key can run again.
Write the race tests first. The decision record's exit criteria are the right list: concurrent duplicates, crash and retry, expiry, response replay and stale-update races, per endpoint, before that endpoint is called released.
Fix the status codes on day one. Changing one later breaks every client that learned the old one.
Key takeaways
- A timeout hides three outcomes. Make retries always safe instead of guessing which one happened.
- The client mints one key per user intent and reuses it on every retry.
- Scope receipts by caller, tenant and operation, and compare a canonical fingerprint so a reused key can't replay the wrong answer.
- Commit the receipt in the same transaction as the first durable effect. Anything else leaves a window where the work happened and the receipt doesn't know.
- Versions stop lost updates between people; keys stop duplicates from one client. Check the receipt first, then the version.
- Expire receipts into tombstones, not deletions, so an old key fails loudly.
Sessions come next, and they use the same trick: refresh token rotation is a conditional update that only one request can win.