Thirteen services shared one coffee-chain platform, and every one of them believed something about an order. Checkout owned the order itself. Loyalty owed the customer points. Inventory had to hold stock. Reporting counted revenue. The moment those four stopped agreeing, no amount of retrying fixed it — because by then nobody could say which of them was right.
This is the failure that pushed me to the transactional outbox, and the one pattern I would keep if I could only keep one.
The write that isn't one write
The naive version looks correct:
await prisma.order.create({ data: order })
await kafka.send({ topic: 'order.placed', messages: [{ value: json }] })
Two writes. Two systems. No transaction spans both.
Everything between those lines is a place to fail. The pod is evicted. The broker is mid-rebalance. The request times out while the broker has, in fact, accepted the message. Postgres commits and the process dies before the second line runs.
You now have an order that exists and an event that does not. Loyalty never credits the points. Inventory never reserves the stock. The customer sees a confirmation and, a week later, an email saying the item was never in stock.
Why a retry does not save you
The usual instinct is to wrap the publish in a retry. It does not help, and it is worth being precise about why.
A retry can only run if the process is still alive. The failures that actually hurt are the ones that take the process with them. And the timeout case is worse than a plain failure: you cannot distinguish "the broker never got it" from "the broker got it and the acknowledgement was lost". Retry the first and you are correct; retry the second and you have published twice.
Moving the publish before the commit only inverts the damage — now you can emit an event for an order that was rolled back, which is harder to detect and much harder to undo.
There is no ordering of two independent writes that makes them atomic. The problem is the second system, not the sequence.
The outbox
So stop writing to the second system.
The event becomes a row in the same database, written in the same transaction as the business change:
BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES (…);
INSERT INTO outbox (aggregate_id, type, payload)
VALUES (…, 'order.placed', '{"orderId": …}');
COMMIT;
Either both rows exist or neither does. That is the whole idea, and it is the only part that has to be exactly right.
A separate relay then reads unpublished rows and sends them to Kafka. If it crashes halfway, the rows are still there and it picks up where it left off. If it publishes and dies before marking the row sent, it will publish that row again on restart.
That last sentence is not a defect. It is the contract: at-least-once delivery.
Publishing without lying about ordering
Two details decide whether this holds up under load.
Partition by aggregate, not round-robin. Every event for one order goes to the same Kafka partition, keyed by the order id. Kafka guarantees ordering within a partition and nothing across partitions. Key on the aggregate and order.placed can never arrive after order.cancelled for the same order. Key on anything else and it eventually will.
Claim rows before sending. Several relay instances will run. SELECT … FOR UPDATE SKIP LOCKED lets each take a distinct batch without blocking the others:
SELECT id, type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 100;
SKIP LOCKED is what makes the relay horizontally scalable. Without it, the second instance waits on the first and you have an expensive way to run one worker.
The consumer's half: an inbox
At-least-once means duplicates are normal, so consumers have to be able to see the same event twice without doing the work twice.
Every consumer keeps an inbox table with a unique constraint on the event id. The insert happens in the same transaction as the side effect:
await tx.inbox.create({ data: { eventId } }) // throws on the second attempt
await tx.loyalty.increment({ customerId, points })
The second delivery violates the unique constraint, the transaction rolls back, and nothing is double-counted. At-least-once delivery, exactly-once effect.
This is where teams usually cut the corner, and it is the half that determines whether the guarantee is real. An outbox without an inbox just moves the duplicate problem one service downstream.
What it costs
It is not free, and pretending otherwise is how people get surprised:
- Latency. Polling adds delay. A one-second interval was invisible to users and cheap on the database; sub-second polling was not worth it. Change data capture removes the poll entirely, at the cost of running Debezium.
- A table that grows. The outbox needs pruning. Published rows older than a few days are deleted on a schedule — keep enough to debug, not enough to matter.
- Ordering is only per aggregate. Anything needing a global order needs a different design. In practice nothing did.
- Payloads are a contract. A row written last week is consumed by code deployed today, so the payload is versioned and consumers ignore unknown fields.
What I would do differently
I would write the inbox first. We built the outbox, watched duplicates arrive, and added deduplication afterwards under time pressure. The two halves are one pattern, and the second half is the one that makes the first half true.
I would also resist making the payload a full snapshot of the aggregate. It is tempting — the consumer needs no follow-up call — but every field becomes a contract you cannot change. Identifiers and the facts that actually changed age far better.
Thirteen services, one source of truth, and the broker never written to directly. That is the whole pattern.
