A request comes in through the gateway. The service behind it calls a second service over gRPC, and today the second one is slow. The gateway runs out of patience first, so the client gets a 504 and leaves. Behind the gateway, the query keeps running, the rows still get written, and a third service may still get a call. All of it is for a client that has already gone.
Nothing crashed, and every line did what it was told. Before you choose between REST, gRPC and Kafka, understand this failure, because each transport has its own version of it.
Why didn't the timeout stop the work?
Here is the usual wrong fix at its smallest. Save it as race.mjs and run node race.mjs:
const slowLookup = () =>
new Promise((resolve) =>
setTimeout(() => {
console.log('lookup finished; nobody is listening');
resolve('stock: 12');
}, 3_000),
);
const giveUpAfter = (ms) =>
new Promise((_, reject) =>
setTimeout(() => reject(new Error(`timed out after ${ms} ms`)), ms),
);
try {
console.log(await Promise.race([slowLookup(), giveUpAfter(1_000)]));
} catch (error) {
console.log(error.message);
}It prints timed out after 1000 ms, then two seconds later lookup finished; nobody is listening. Promise.race picks which result you look at; it doesn't stop the loser. A timeout is the waiter's decision. Work stops only when someone tells the worker.
gRPC makes this worse by default. Its deadlines guide is blunt: without a deadline, a client can wait for a reply more or less forever. The gateway has a clock of its own. Kong, the gateway on this platform, waits 60 seconds for an upstream by default. If a connection or the response headers time out, it also retries idempotent methods (GET, PUT, DELETE) up to five times (Kong proxying docs). So the outer layer gives up after a minute and may send the request again. The inner call has no deadline, so it never gives up.
Picture a newsroom at six in the evening. A reporter phones a source for a quote, and the source says "let me check the files". At press time the editor cuts the paragraph and the paper goes to print. Nobody phones the source back, so the source keeps digging all evening for a story that has already run.
What are the three ways microservices talk?
A newsroom has three kinds of conversation, and each one maps to a transport.
The printed paper is public and has a fixed format. Once it's out you can't take it back; you can only print a correction tomorrow. That's the public API: REST/JSON through one gateway. A mobile app released months ago is a reader holding an old edition, still calling the API as it looked when the app shipped.
The phone call before the deadline is a question someone needs answered before they can carry on. Both people must be on the line at once, and the call has a time limit. That's gRPC between services.
The wire bulletin announces something that has already happened. The agency doesn't wait to see who reads it, and each paper picks it up at its own pace. Now and then a bulletin arrives twice. A correction goes out as a new bulletin and never edits the old one. That's a Kafka event.
A fourth thing matters more than the other three: the archive. Before phoning anyone, a good reporter checks the clippings. If yesterday's figure is good enough, there's no call to make. For a service, the archive is a local projection: a table built from other services' events.
REST, gRPC or Kafka: how do you choose?
I ask three questions, in this order, and stop at the first yes:
- Is the other end outside the platform? Then use REST through the gateway. Browsers, mobile apps and partner systems never get a gRPC port or a Kafka topic.
- Did a service just change its own state? Then it publishes an event after the change commits. No consumer can veto something that has already happened.
- Does it need an answer before it can safely continue, and would a stale local copy be unsafe? Only then does it make a gRPC call. If the archive will do, read the archive.
The third question matters most. Every synchronous call needs two services alive at the same moment, so the platform's rule is that a command calls another service only when it can't safely continue with a local projection.
When one kind of order is created, the order service asks inventory over gRPC whether there's enough stock for those lines right now, with a 2-second deadline. A projection a few seconds old could oversell. If inventory doesn't answer, order creation fails closed with a 503 instead of guessing. That has a real cost: when inventory is down, so is creating that kind of order.
A customer's points balance goes the other way. It comes from a local projection, and the response says how old the data is: an asOf timestamp, plus a freshness flag that switches to STALE after 60 seconds. It's the archive, with the date printed on the clipping.
What should an API gateway own?
The gateway owns whatever is true of every request, whichever service handles it: routing, TLS termination, CORS, rate limits and a correlation ID. It also deletes every header a client could use to lie about who it is. Here is the global plugin block from the declarative Kong config, trimmed:
plugins:
- name: correlation-id
config:
header_name: X-Correlation-ID
generator: uuid
- name: request-transformer
config:
remove:
headers:
- X-User-Id
- X-Session-Id
- X-Roles
- X-Permissions
- X-Scopes
- X-Consumer-IDIf a service ever reads X-Roles, that header had better come from inside; stripping it at the edge makes that true by construction. Everything else belongs to the service that owns the data: business validation, authorization policy, persistence and the response body.
Authentication at the edge, authorization in the service
On protected routes, Kong's JWT plugin rejects a token that is missing, badly signed or expired. That's the press-pass check at the front door, but a press pass doesn't let you read the editor's files.
The gateway's check is stateless: it can't know the session was revoked a minute ago. So each service sends the bearer token to the auth service over gRPC, and auth checks the session row, the account state and whether the token's roles still match, then returns the verified principal. That's question three again, since stale identity is unsafe. The price: when auth is down, every protected route is down.
Inside the service, a guard does the rest. A route that forgets to declare itself public falls back to "authenticated", never to "open", and permission codes like <SERVICE>:<RESOURCE>_<ACTION> are checked by the service that owns the resource.
Why did the edge stop translating JSON into gRPC?
The first design had Kong's grpc-gateway plugin transcode browser JSON into gRPC calls, using annotations in the .proto files. It lasted four days. Making it behave took two custom Lua plugins, one to reject unknown JSON fields and one to turn protobuf's null-value objects back into a plain null. Every quirk of protobuf's JSON mapping was becoming a quirk of the public API.
The replacement is simpler. Each of the thirteen services boots one NestJS hybrid application: an HTTP listener and a gRPC listener in the same process, both calling the same application layer. Kong just proxies HTTP. That's one translation fewer, and the public contract is JSON written for clients, not generated from an internal schema.
One error shape for two transports
Every public error is meant to share one envelope: { statusCode, errorCode, message, details?, timestamp, requestId }. The trick is a shared error type that belongs to neither transport. A use case throws an error with a kind, the kinds deliberately copy gRPC's status codes, and each boundary translates:
| Error kind | gRPC status | HTTP status |
|---|---|---|
NOT_FOUND |
NOT_FOUND |
404 |
FAILED_PRECONDITION |
FAILED_PRECONDITION |
422 |
ABORTED |
ABORTED |
409 |
DEADLINE_EXCEEDED |
DEADLINE_EXCEEDED |
504 |
Raw framework, validator, ORM and gRPC messages never reach a client. The gap: one service still lets the framework's default filter answer some HTTP failures, so clients can see two shapes. Documented, but still a bug from the client's side.
How do you make a gRPC call between services safe?
Every internal call goes through a small helper that signs the metadata and computes an absolute deadline. This is a trimmed client, with the package scope renamed:
import { ServiceUnavailableException } from '@nestjs/common';
import { invokeSignedGrpcAsync } from '@platform/common';
import {
CHECK_COVERAGE_DEADLINE_MS,
CHECK_COVERAGE_METHOD,
checkCoverageRequestSchema,
checkCoverageResponseSchema,
hashCheckCoverageRequest,
} from '@platform/contracts';
const COVERAGE_UNAVAILABLE = 'INVENTORY_COVERAGE_UNAVAILABLE';
async coveredItemIds(itemIds: readonly string[]): Promise<readonly string[]> {
const request = checkCoverageRequestSchema.parse({ itemIds });
try {
const response = await invokeSignedGrpcAsync(
{
serviceId: this.config.serviceId,
method: CHECK_COVERAGE_METHOD,
payloadDigest: hashCheckCoverageRequest(request),
sharedSecret: this.config.inventorySharedSecret,
deadlineMs: CHECK_COVERAGE_DEADLINE_MS,
},
({ metadata, deadline }) => this.invoke(request, metadata, deadline),
);
const parsed = checkCoverageResponseSchema.parse(response);
const askedFor = new Set(request.itemIds);
if (parsed.coveredItemIds.some((id) => !askedFor.has(id))) {
throw new ServiceUnavailableException(COVERAGE_UNAVAILABLE);
}
return parsed.coveredItemIds;
} catch {
throw new ServiceUnavailableException(COVERAGE_UNAVAILABLE);
}
}Four details do the work.
Put the deadline in the contract
CHECK_COVERAGE_DEADLINE_MS sits in the shared contracts package, right next to the method name. (The Nx monorepo article explains that package.) Keeping it there makes "how long is too long" part of the contract, not one client's guess. In this codebase the values range from 500 ms to 3 seconds.
Notice the direction: the gateway waits 60 seconds, internal calls give up after 0.5 to 3. Budgets shrink as you move inward; an inner call that outlives its caller is the opening bug again. Not every deadline has moved into the package yet; several still live inside one client.
A deadline stops the waiting, not the work
On the client side, invoke wraps the call in an Observable. If the subscriber leaves before the call settles, the teardown calls call.cancel(). The deadline also travels with the request, so the server-side call gets cancelled when it passes.
Read the gRPC guide closely, though. Stopping the work is the server application's job. grpc-js sets call.cancelled and emits a 'cancelled' event, but a NestJS handler that never checks keeps running its queries. None of the handlers here check yet. The fix is cheap:
import { status, type Metadata, type ServerUnaryCall } from '@grpc/grpc-js';
import { GrpcMethod, RpcException } from '@nestjs/microservices';
@GrpcMethod('InventoryService', 'CheckCoverage')
async checkCoverage(
request: CheckCoverageRequest,
_metadata: Metadata,
call: ServerUnaryCall<CheckCoverageRequest, CheckCoverageResponse>,
): Promise<CheckCoverageResponse> {
const levels = await this.stock.loadLevels(request.itemIds);
if (call.cancelled) {
throw new RpcException({ code: status.CANCELLED, message: 'caller gave up' });
}
return this.coverage.evaluate(levels, request.itemIds);
}The check won't interrupt a query that's already running, but it stops the handler from starting the next step. A Postgres statement_timeout caps any single query.
Writes are harder. Say the deadline passes just after the commit: the caller sees DEADLINE_EXCEEDED for work that actually happened. It can't tell "never happened" from "happened after I left". That's exactly what an idempotency key is for.
Sign the caller, and still encrypt the channel
The helper attaches the caller's service ID, method, timestamp, a nonce, the request and correlation IDs and a SHA-256 digest of the payload, then signs it all with HMAC-SHA256 and a secret the two services share. The receiver checks caller and method, rejects timestamps more than 60 seconds off and compares signatures in constant time; services with a nonce table also reject a nonce seen in the last two minutes. That stops forged and replayed calls.
It doesn't stop anyone from reading them. Signing is not encryption, and in this prototype the gRPC channels inside the private network are still plaintext. TLS is on the list, but it isn't done.
Validate the answer too
The other service is a separate codebase, deployed on a different day. The client parses the response with the shared schema and checks that every returned ID is one it asked about. Every failure collapses into the same well-defined 503, and retries are bounded and limited to safe or idempotent operations.
When should a service publish a Kafka event?
Only after the owning use case has committed the change and decided to announce it. As the decision record puts it, a library, topic or schema doesn't prove an HTTP write emits an event; a traced producer and consumer does.
What makes "after commit" true is the transactional outbox: the event is a row written in the same transaction as the state change, and a relay publishes it later. Delivery is at-least-once, so by convention a consumer records each event in an inbox table in the same transaction as its own change, and commits the Kafka offset only afterwards.
Topics are named platform.<service>.<aggregate>.<past-tense-event>.v1, with platform standing in for the real prefix. Every event carries the same envelope:
{
"eventId": "0190f3a2-7c1e-7d4b-9a2f-3b8c6d1e5f70",
"eventType": "platform.inventory.receipt.confirmed.v1",
"schemaVersion": 1,
"aggregateId": "0190f3a1-2b6d-7e0a-8c4f-9d2e1a7b3c58",
"aggregateVersion": 4,
"occurredAt": "2026-07-20T08:15:30.000Z",
"producer": "inventory",
"correlationId": "0190f3a1-f0a3-7b21-a6d9-4e8c2b1f7a05",
"causationId": "0190f3a1-e9b7-7c02-b1e4-7f3a5d2c8e91",
"payload": { "status": "CONFIRMED" }
}Four rules keep the bulletins honest:
- Key by aggregate ID. Kafka only orders messages within a partition. Today every topic has one partition on a single broker. That makes ordering free, but it limits each topic to one active consumer per group. Keying by aggregate keeps ordering correct on the day someone adds partitions, and
aggregateVersionstill lets a consumer spot a stale event. - Only additive changes within a version. Adding a new optional field is fine. Renaming a field means a
.v2topic and a migration window. - Bounded retries, then a dead-letter topic, with alerts and an authorized replay tool.
- Carry the thread.
correlationIdfollows one user request through the gateway, the gRPC calls and every event it caused.causationIdnames the direct cause. An optionalpayloadHashexposes the sameeventIdarriving with a different body.
How does each transport fail?
| Layer | Characteristic failure | What contains it |
|---|---|---|
| Gateway | Business rules creep in, one small check at a time | A written list of what the edge owns |
| REST contract | A change breaks an app released months ago | Additive changes, explicit versions |
| gRPC | Hung calls, retry storms, one outage spreading to callers | Deadlines, cancellation, bounded retries |
| Kafka | Lost events, duplicates, reordering, a poison message | Outbox, inbox, keyed partitions, dead letters |
gRPC fails through coupling: three synchronous calls in a row need four services up at once, like a story that runs only if all three sources pick up. Kafka fails quietly: a dual write loses the bulletin whenever the process dies between database and broker, and nobody notices until a projection is wrong.
What do three transports cost?
Three transports mean three things to operate, three kinds of contract and three kinds of test. gRPC needs a policy for deadlines, retries, TLS and trace context; Kafka needs partitioning, retention, replay, monitoring and idempotent consumers. The decision record deliberately won't call either one faster or more reliable in general.
Evidence matters too. Only the auth path has an end-to-end run on record (HTTP through the gateway, plus gRPC readiness and profile calls). Cross-service retries, dead-lettering and duplicate handling are coded but not verified, and I'd rather say so than imply a tested guarantee.
With three services and one team, three transports are probably too many: REST between services and a database-backed job queue will carry you a long way. The split pays once services change at different speeds and coordinating a release costs more than running a broker. If you can't name a consumer that needs to hear about a change without the producer knowing, you don't need an event yet.
What I'd do differently
- Check cancellation on the server from the first handler. A client deadline is half a feature; the check and a statement timeout belong in the service template.
- Generate client types from the
.protofiles. Hand-written gRPC interfaces are where drift starts. - Pass the correlation ID on every hop. Only one of the helper's seven callers passes one; the rest mint a fresh ID and cut the trace at the first internal call.
- Lint topic names in CI. Event names mix
snake_caseandkebab-case, and dead-letter topics use two suffixes. - Close the asymmetries. Two read-only integrations call another service's HTTP endpoint unsigned. Documented as intentional, still two security models.
- Change the decision record in the same commit as the config. Ours still lists "gRPC transcoding" as a gateway job. When a plan and the running config disagree, trust the config, then fix the plan before the next reader designs against a gateway that no longer exists.
Key takeaways
- A timeout stops the waiting, not the work. Give every gRPC call a deadline, cancel on teardown, and check for cancellation on the server.
- Ask in order: is the other end outside the platform (REST)? Did a service just change its own state (Kafka)? Does it truly need a fresh answer (gRPC)? If none of these applies, read a local projection.
- Time budgets shrink inward, so every inner deadline must be shorter than the outer timeout.
- The gateway authenticates and strips spoofable headers. The owning service authorizes.
- Clients see plain JSON, never protos or topics.
- Events go out after commit, through an outbox, keyed by aggregate, to consumers that expect duplicates.
Next in the series: database per service and sagas, where one user action must change state in several services and no transaction is big enough to hold it.