Here is a bug that passes every test you will write for it.
A request creates an order. The service commits the row to Postgres, then publishes order.created to Kafka so that stock gets reserved and a confirmation goes out. In tests, both lines run. Then a deploy restarts the container between them. The order exists, but the event never leaves. Nothing downstream will ever hear about it, and no log says anything went wrong, because as far as the code knows nothing did.
The standard answer is the transactional outbox, and the outbox table is the easy 10%. Lost and duplicated events come from the other 90%: who sends each row, what happens when a sender stalls, and how a consumer spots a message it has already handled. Below, I build it up one failure at a time, with real numbers from a platform of thirteen NestJS services, twelve of which have an outbox table. (Why events go over Kafka at all: REST, gRPC or Kafka.)
Keep one picture in mind: registered mail. The clerk writes the letter into a ledger before it leaves the counter, and the recipient signs for it on arrival. The ledger means a letter can't vanish without a trace. The signature means a letter delivered twice gets noticed the second time. You need both.
What is the dual write problem?
A dual write means writing to two systems with no transaction covering both:
await prisma.order.create({ data: order }); // 1. Postgres commits
await producer.send({ // 2. Kafka, maybe
topic: 'order.created.v1',
messages: [{ key: order.publicId, value: JSON.stringify(event) }],
});Each line is atomic on its own, but the pair isn't. Between them sit a process that can be killed, a broker that can be mid-election, and a network that can drop the acknowledgement after the broker has already written the message. Any of these leaves Postgres and Kafka disagreeing about what happened.
Why doesn't retrying the publish fix it?
The reflex is to wrap line 2 in a retry. But a retry runs inside the process, and the failures that matter take the process with them: an OOM kill, a host reboot, a deploy.
A timeout is worse: you can't tell "the broker never got it" from "the broker got it and the ack was lost". Retrying is right in the first case and publishes twice in the second. Retries turn lost events into duplicates. That's progress, not atomicity.
Publishing first only flips the damage: a failed commit leaves an event announcing an order that was rolled back. Kafka's idempotent producer doesn't help; it deduplicates its own retries within one session and knows nothing about your database.
No ordering of two independent writes makes them one write. So stop doing two.
How does the transactional outbox work?
Write the event into the same database, in the same transaction as the business change. The event becomes a row:
CREATE TABLE outbox_events (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
event_id uuid NOT NULL UNIQUE,
aggregate_id uuid NOT NULL,
aggregate_version int NOT NULL,
event_type text NOT NULL,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'PENDING', -- PUBLISHING, PUBLISHED, DEAD
attempt_count int NOT NULL DEFAULT 0,
next_attempt_at timestamptz NOT NULL DEFAULT now(),
claim_token uuid,
claim_expires_at timestamptz
);await prisma.$transaction(async (tx) => {
const order = await tx.order.create({ data: orderData });
await tx.outboxEvent.create({
data: {
eventId: createUuidV7(),
aggregateId: order.publicId,
aggregateVersion: order.version,
eventType: 'order.created.v1',
payload: { orderId: order.publicId },
},
});
});Both rows commit or neither does. That's the ledger entry at the counter: the letter is on record before it leaves.
A separate process, the relay, reads committed rows and sends them to Kafka. If it crashes, the rows wait. If it sends a row and dies before marking it sent, it sends that row again after restart. That isn't a bug to fix later; it's the contract. The outbox gives you at-least-once delivery, and the rest of this article is about living with "at least".
How should a relay claim rows?
With one relay, claiming is trivial. You run two for availability, and now both see the same pending rows.
What does FOR UPDATE SKIP LOCKED do?
FOR UPDATE locks the rows a query returns. SKIP LOCKED tells other queries to step over locked rows instead of waiting. Try it in two psql sessions:
-- session A
BEGIN;
SELECT id FROM outbox_events WHERE status = 'PENDING'
ORDER BY id LIMIT 2 FOR UPDATE SKIP LOCKED; -- returns 1, 2
-- session B, while A is still open
BEGIN;
SELECT id FROM outbox_events WHERE status = 'PENDING'
ORDER BY id LIMIT 2 FOR UPDATE SKIP LOCKED; -- returns 3, 4, without waitingWithout SKIP LOCKED, B waits for A and you've built an expensive way to run one worker. Without FOR UPDATE, both get rows 1 and 2.
Why not hold the lock while sending?
Because the send is a network call. An open transaction during it pins a database connection and holds row locks for as long as a stalled send lasts. So the lock only decides who gets a row; a lease decides how long they keep it. The order service claims and leases in one statement and commits immediately:
WITH candidates AS (
SELECT id FROM outbox_events
WHERE (status = 'PENDING' AND next_attempt_at <= now())
OR (status = 'PUBLISHING' AND claim_expires_at <= now()) -- lease ran out
ORDER BY id
LIMIT 50
FOR UPDATE SKIP LOCKED
)
UPDATE outbox_events AS e
SET status = 'PUBLISHING',
claim_token = $1, -- fresh UUID per poll
claim_expires_at = now() + interval '30 seconds',
attempt_count = e.attempt_count + 1
FROM candidates
WHERE e.id = candidates.id
RETURNING e.*;It polls every 5 seconds and takes up to 50 rows with a 30-second lease. A row left in PUBLISHING past its lease becomes claimable again, which is what saves you when a relay is killed mid-batch.
What happens when a relay stalls mid-send?
Leases bring a new failure. Relay A claims a row at t = 0, and its send hangs for 40 seconds. At t = 30 the lease expires; relay B claims the row, sends it and marks it published. At t = 40, A's send fails and A writes "reschedule this row". A has just put a published event back in the queue.
The claim token is the numbered dispatch slip a courier has to hand back. Every write after the claim checks it:
UPDATE outbox_events
SET status = 'PUBLISHED', claim_token = NULL
WHERE event_id = $1
AND claim_token = $2 -- only the current holder may finish
AND status = 'PUBLISHING';A's late write now matches zero rows. But notice what the token doesn't do: if A's send had succeeded at t = 40, Kafka would have the event twice. This is called fencing: every write after the claim must prove the writer still holds it. Fencing stops stale writers from corrupting state; it can't un-send a message. Duplicates remain the consumer's problem.
When should an event stop retrying?
A failed send goes back to PENDING with a later next_attempt_at. In the order service the delay starts at 1 second and doubles, with ±20% jitter so relays don't retry in lockstep and a 15-minute cap. After 8 attempts an ordinary event is dead-lettered.
Do the arithmetic before trusting that. Delays of 1, 2, 4, 8, 16, 32 and 64 seconds add up to about two minutes; the cap is never reached. The Kafka client retries inside each attempt, but a broker that's down for fifteen minutes still turns a queue of events into dead letters. Budget retries in time ("survive a 30-minute outage") and derive the count.
Dead letters travel by the same registered mail: the row is marked DEAD, and the same transaction inserts a new outbox row describing the failure. Its id is createDerivedUuidV7(sourceEventId, deadLetterEventType): the source's timestamp bits plus a SHA-256 of the source id and the dead-letter event type. Same input, same id, so a dead-letter step that ran twice would collide on the unique event_id rather than announce the failure twice.
An unmapped event type or a payload that fails envelope validation is dead-lettered at once. Retrying either just fails eight times.
Does keying by aggregate keep events in order?
Kafka orders messages within a partition, nowhere else. Sending with key: aggregateId puts every event for one order on one partition, so order.cancelled can't overtake order.created inside Kafka.
The relay can reorder before Kafka sees anything, though. The order service sends each batch in parallel. If version 1 fails and is rescheduled while version 2 succeeds, version 2 goes first. The key was right and the order is still wrong.
Two services close this gap at claim time, refusing a row while an earlier version of the same aggregate is still waiting:
AND NOT EXISTS (
SELECT 1 FROM outbox_events earlier
WHERE earlier.aggregate_id = candidate.aggregate_id
AND earlier.aggregate_version < candidate.aggregate_version
AND earlier.status IN ('PENDING', 'PUBLISHING')
)Then decide what a dead predecessor means. One service leaves dead rows out of the check, so later events flow and consumers see a gap. The other blocks on anything not yet sent, so one failed event freezes that aggregate's stream until someone steps in. Both are defensible. Not deciding isn't.
Why does every consumer need an inbox?
At-least-once means some messages arrive twice. The consumer also has a dual write of its own: it commits its database transaction, then commits the Kafka offset. Die in between and the broker redelivers. So the consumer keeps a signature book, the inbox, checked in the same transaction as the side effect.
Why isn't "check, then insert" enough?
This looks right:
const seen = await tx.inboxEvent.findUnique({ where: { eventId } });
if (!seen) {
await applySideEffect(tx, event);
await tx.inboxEvent.upsert({ where: { eventId }, create: row, update: row });
}Two concurrent deliveries both read null, both apply the side effect, and the upsert turns the second insert into an update instead of an error. A unique constraint only protects you when the losing writer fails. A rebalance or a retry worker beside the live consumer is enough to cause this. The decision record for the pattern still lists this race in inbox handling.
The simplest fix is to claim first:
INSERT INTO inbox_events (event_id, payload_hash)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING;
-- 1 row: first delivery; apply the side effect in this transaction
-- 0 rows: already signed for; compare payload_hash, then skipA concurrent insert of the same key waits for the first transaction, then does nothing. One service goes another way: its inbox step runs at SERIALIZABLE and retries on a unique violation or serialization failure, up to five attempts. Either way, no constraint can take back a side effect that left the database (an email, an HTTP call). Make that another outbox row.
Which key should the inbox deduplicate on?
Different duplicates leave different fingerprints:
| Key | Catches | Misses |
|---|---|---|
event_id |
the relay sending a row twice | the same fact re-emitted with a new id |
| topic + partition + offset | redelivery after a lost offset commit | one row published twice (two offsets) |
| producer + aggregate + version + type | the same fact under any id | nothing above, but needs versioned events |
The two most thorough inboxes in the platform enforce all three as unique keys. It's an idempotency key, applied to messages instead of HTTP requests.
What if the same id arrives with a different payload?
Then it isn't a duplicate; it's a producer bug or a replay that changed data. The order service's envelopes carry a payload hash. A careful consumer compares it with the hash stored for that id and dead-letters a mismatch instead of skipping it. Same tracking number, different contents: don't sign.
How do you handle stale and early versions?
Track the last aggregate_version processed per aggregate:
- version ≤ last: stale or duplicate. Record and skip.
- version = last + 1: apply.
- version > last + 1: something is missing. Park it.
How you park matters. Stopping the partition until the gap fills is simple, but it deadlocks if the missing event sits behind you in the same partition, which is exactly what relay reordering produces. Storing the early event, committing its offset and retrying it from a worker doesn't. Both shapes exist in the platform.
The order service parks early saga outcomes, retries them up to five times, then rejects the outcome, moves the saga to MANUAL_RECOVERY and emits a dead-letter event for a human. The saga article explains why that state exists.
How long should outbox and inbox rows live?
Every business write adds an outbox row, updated at least twice more on its way to PUBLISHED. Keeping them forever is the default, and it's wrong. Only one service has a retention worker so far: it deletes published outbox rows and processed inbox rows older than 30 days by default (configurable from 7 to 365), 500 rows per batch. Elsewhere the tables just grow. Two rules for setting it:
- Inbox retention must outlast the longest redelivery window: broker retention plus however far back you might replay. Delete a signature and the next copy of that letter gets signed again.
- Don't derive checkpoints from rows you delete. If "last processed version" is
MAX(version)over inbox rows, retention quietly deletes it, and the next event for a quiet aggregate looks like a gap. Keep checkpoints in their own table. The service with retention computes its version check this way, which is how I noticed.
Payloads are data too: the order service encrypts a personal address inside the event payload that carries it, instead of leaving it in clear text in a table nobody prunes.
What it costs
- Latency. A 5-second poll means up to 5 seconds before an event leaves, more while backing off. Change data capture from the WAL removes the poll at the price of another system to run.
- Write amplification. An extra insert plus claim and publish updates per business write: more WAL and vacuum on the database that serves your requests.
- Code. Twelve relays. They share helpers (UUIDs, backoff, hashing) but not the relay or inbox, and they differ: batch or one row at a time, with or without predecessor checks.
- Ordering is per aggregate, and only if you enforce it.
- Payloads are contracts. Rows written before a deploy are read by code after it, so envelopes carry a
schemaVersionand consumers tolerate unknown fields.
What I'd do differently
Build the inbox first. The outbox feels like the solution, so it gets built first, and deduplication gets added under pressure once duplicates appear. Duplicates aren't an accident; they're the contract.
One relay, one inbox, one test suite. I'd move both into the shared library and test them against a real Postgres: two relays, kill one mid-send, assert nothing is lost and duplicates are caught downstream. Today one relay's unit test checks that its claim SQL contains FOR UPDATE SKIP LOCKED, which is not the same thing. The decision record is honest about it: a target pattern with a partial prototype, duplicates still possible, concurrency and crash recovery unproven. Leases and claim tokens have landed since, but written isn't proven.
Decide ordering on purpose, per stream: does order matter, does a dead predecessor block, and who gets paged.
Budget retries in minutes, not attempts.
Key takeaways
- Retrying or reordering can't make a dual write atomic. Write the event in the same transaction, to the same database.
SKIP LOCKEDdecides who claims a row, a lease decides how long, and a claim token stops a stale holder from overwriting the result.- Fencing prevents corrupted state, not duplicate sends. The consumer's inbox makes processing effectively-once.
- Claim inbox entries with an insert that fails or does nothing on conflict; check-then-upsert lets duplicates through.
- Keying by aggregate doesn't guarantee order if the relay can reorder. Enforce it at claim time or with versions at the consumer.
- Retention, retry budgets and dead letters are part of the pattern, not cleanup for later.
Every event here carries a UUIDv7 minted by the application, and each dead letter's id is derived from its source. Why the app mints ids instead of the database is the next decision: UUIDv7 vs bigint primary keys.