Ask a chat assistant that talks straight to a language model: "Can I get a refund after 30 days?" It answers at once, politely, with a specific number of days and a condition or two. The answer sounds right. It also appears in no document the company has ever published.
The model wasn't lying. It did what it was trained to do: continue the conversation with text that is likely to come next. For small talk, that's the feature. For a policy question it's a liability, because the reader can't tell a remembered fact from a fluent guess, and neither can the model.
I had to fix exactly this in a platform of thirteen NestJS services. One of them runs a chat assistant that answers questions from a handful of policy documents and a product catalog. The fix is retrieval-augmented generation (RAG), and the vector search turned out to be the easy part. What matters is three smaller decisions: what happens when nothing matches, how an answer shows where it came from, and how the index stays current. All of it runs inside Postgres. Everything below comes from the repository, with the numbers exactly as they appear in the code, including the ones I haven't tuned yet.
Keep one picture in mind: an open-book exam where every answer must cite the page. The book makes a good answer possible. The citation rule makes it checkable. And a good student who finds no page on the question writes "not covered in the material" instead of improvising. RAG hands the model the book. Most of this article is about the other two rules.
Why does an LLM answer questions it has no source for?
A model's weights are a compressed memory of its training text. Your refund policy isn't in there. If anything is, it's someone else's refund policy, or an older version of yours. Asking the model directly is a closed-book exam: it will write something, because writing something is all it can do. A prompt that says "be accurate" changes the tone, not the knowledge.
The obvious first fix is to paste every policy into the system prompt. To be fair, with five short documents that works, and it's simpler than anything below. I didn't stop there for three reasons. The same assistant also searches a product catalog, which is larger and changes all day. Every turn would pay for every document in tokens. And the answer still couldn't say which document it used.
What does retrieval-augmented generation actually change?
RAG means one thing: before the model answers, look up the relevant text and give it to the model as evidence. The closed-book exam becomes an open-book one.
Here is the version most tutorials start with:
const hits = await vectorSearch(question, { topK: 3 });
const answer = await llm.chat([
{
role: 'system',
content: `Answer using this context:\n${hits.map((hit) => hit.body).join('\n---\n')}`,
},
{ role: 'user', content: question },
]);Now ask it "Can I pay in instalments?" when no document mentions instalments. topK: 3 still returns three documents, because a nearest neighbour always exists. It returns the three least distant documents, say the refund policy, the data-export policy and the account-deletion policy. The model now holds official-looking context that is off-topic, and the system prompt has told it to use that context.
This is the open-book student who flips to the closest page and writes an answer anyway. Naive RAG doesn't remove hallucination. It can make a hallucination more convincing, because now it quotes real text.
So the real questions are: how does the search decide that nothing is relevant, and how does that decision reach the user intact?
Why keep the vectors in Postgres instead of a vector database?
The documents already lived in a Postgres database owned by the assistant's service (one database per service; the reasoning is in database per service and sagas). The pgvector extension adds a vector column type and distance operators. That bought three things:
- One transaction. A document's text, embedding, published flag and version change together. There's no second store that can drift.
- Filters are
WHEREclauses. Whether a document is published, or whether an item is available, is a column you filter on. Similarity never gets to guess it. That was an explicit rule in the original design. - Full-text search is in the same engine, so hybrid search is a single query.
What you give up is a dedicated engine's tooling for large-scale approximate search. For a corpus counted in documents plus one catalog, that trade is easy.
The policy table, with the names made neutral:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE policy_documents (
id BIGSERIAL PRIMARY KEY,
doc_key VARCHAR(80) NOT NULL UNIQUE, -- the citation
title_vi VARCHAR(200) NOT NULL,
title_en VARCHAR(200) NOT NULL,
body_vi TEXT NOT NULL,
body_en TEXT NOT NULL,
search_text TEXT NOT NULL,
search_vector TSVECTOR GENERATED ALWAYS AS (to_tsvector('simple', search_text)) STORED,
embedding vector(768),
embedding_model VARCHAR(120),
is_published BOOLEAN NOT NULL DEFAULT TRUE,
version INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX policy_documents_search_idx
ON policy_documents USING GIN (search_vector);
CREATE INDEX policy_documents_embedding_idx
ON policy_documents USING hnsw (embedding vector_cosine_ops)
WHERE embedding IS NOT NULL;A few choices here are deliberate:
- One row holds both languages.
search_textconcatenates the Vietnamese title and body with the English ones (NFKC-normalised), so a question in either language can match. 'simple', not a language configuration. Postgres ships no Vietnamese stemmer.simplelowercases and splits on word boundaries, with no stemming and no stop words. It also keeps diacritics, so a query typed without accents won't keyword-match accented text. That's a known gap, covered later.embeddingis nullable. A document can exist, and be found by keywords, before it has a vector.- 768 dimensions come from a hosted embedding model (Gemini's
gemini-embedding-001, with output dimensionality set to 768). Documents are embedded with the "retrieval document" task type and questions with "retrieval query", so the model knows which side of the search each text sits on.
How does hybrid search combine pgvector and full-text search?
Back to the exam. A student has two ways to find the right page. The index at the back of the book finds exact words: an error code, a product name, "30 days". Skimming for meaning finds the passage that says "money back" when the question said "refund". Each fails where the other works: embeddings blur exact tokens together, and keywords miss paraphrases. Hybrid search uses both.
Here is the query, close to how it appears in the repository:
SELECT doc_key, title_vi, title_en, body_vi, body_en,
( 0.7 * COALESCE(1 - (embedding <=> $2::vector), 0)
+ 0.3 * ts_rank_cd(search_vector, websearch_to_tsquery('simple', $1))
)::double precision AS score
FROM policy_documents
WHERE is_published
AND (
search_vector @@ websearch_to_tsquery('simple', $1)
OR (embedding IS NOT NULL AND (embedding <=> $2::vector) <= 0.75)
)
ORDER BY score DESC, doc_key ASC
LIMIT 3;Read it in two halves.
The WHERE clause is the gate. A row becomes a candidate only if its text matches the keywords or its vector lies within cosine distance 0.75 of the question. <=> is pgvector's cosine distance: 0 means the same direction, 2 means opposite, and 1 - distance is cosine similarity. Everything that fails the gate is gone. This is what makes an empty result possible, and an empty result is what makes "I don't know" possible.
The SELECT and ORDER BY only rank. The score blends 70% vector similarity with 30% keyword rank. COALESCE(…, 0) means a row without a vector competes on keywords alone. The doc_key tie-breaker makes the order deterministic: the same question gives the same documents in the same order, which you can test. LIMIT 3 caps how much the model reads.
Where do 0.7, 0.3 and 0.75 come from?
Judgment, not measurement. The repository has no evaluation set, so nothing shows these are the right values. The blend also has a known flaw. Without a normalisation flag, ts_rank_cd isn't bounded to 0–1, and its values sit on a different scale from cosine similarity. So "70% meaning, 30% keywords" describes the coefficients, not what the arithmetic actually weighs. The 0.75 gate is loose, too: it admits anything with similarity of at least 0.25.
Both have standard fixes, and I list them at the end. For a corpus this small, a loose gate mostly costs an extra off-topic document now and then. The typed tool below keeps that from turning into a confident wrong answer.
Why does the keyword side want keywords, not questions?
websearch_to_tsquery joins unquoted words with AND. Feed it "how long do refunds take" and a document must contain all five words to match, and since simple drops no stop words, "how" and "do" count. So a full question rarely matches on keywords at all.
That's why the model writes the search query, not the user. The tool's input description asks for a policy question or keywords, and its example is a two-word phrase. The model reads the user's long question and passes a short phrase to the tool. The LLM is the query rewriter, and the vector side catches the paraphrases that keywords miss.
Does the HNSW index help this query?
No, and it's worth knowing why. pgvector uses an HNSW index for queries shaped like ORDER BY embedding <=> $q LIMIT n. This query orders by a blended score and filters with a distance threshold inside an OR, so Postgres computes the distance for every published row. With a handful of documents that costs nothing. At a million rows you'd split it: one indexed nearest-neighbour query, one indexed full-text query, then fuse the two candidate lists.
What happens when the embedding API is down?
The query has to be embedded on every question, which puts an external API on the hot path. The embedding client in the repository times out each attempt after 10 seconds, makes up to three attempts (one per configured API key), rejects a vector of the wrong dimension, and returns null instead of throwing. The search branches on that (simplified):
const vector = await embeddings.embedQuery(query); // readonly number[] | null
const rows = vector
? await documents.searchHybrid(query, vector, MAX_RESULTS)
: await documents.searchKeywordsOnly(query, MAX_RESULTS);The keyword-only query is the same SQL minus the vector terms. Quality drops (paraphrases stop matching) but the feature keeps working. The direction of the failure matters: with fewer matches, the assistant says "I don't know" more often. It becomes more cautious, never more inventive.
How do you make a RAG system say "I don't know"?
The gate lives in retrieval, not in the model. By the time the model sees anything, the system has already decided whether there's evidence. If there isn't, the model gets a typed fact saying so.
There are two different empty outcomes, and mixing them up is the classic mistake:
- Nothing matched. The search ran and found nothing relevant. The true sentence is "there's no policy on that".
- Couldn't search. The database or the network failed. The true sentence is "I can't check right now".
The wrong version catches every error and returns []. That turns an outage into a factual claim: "we have no such policy". The repository keeps them apart:
async getPolicy(input: unknown): Promise<PolicyToolOutput> {
const { query } = policyToolInput.parse(input);
try {
const vector = await this.embeddings.embedQuery(query);
const rows = await this.documents.search(query, vector, MAX_RESULTS);
if (rows.length === 0) {
return policyToolOutput.parse({
available: false,
policies: [],
reason: 'no_policy_match',
});
}
return policyToolOutput.parse({ available: true, policies: rows.map(toSummary) });
} catch (error: unknown) {
if (error instanceof ToolOutputValidationError) throw error; // a bug, not an outage
return policyToolOutput.parse({
available: false,
policies: [],
reason: 'policy_unavailable',
});
}
}The system prompt carries the other half: never invent policy content, and when the tool reports no matching data, say so plainly and point the user to support. The server's fixed English copy follows the same split. No match: "I could not find a policy matching that question." Outage: "Policy information is currently unavailable. Please try again later." Two empty results, two different true sentences. Unglamorous, and correct.
Why make retrieval a tool with a strict schema?
You can wire retrieval in two ways: always search before every turn, or expose search as a tool the model calls. This assistant has several tools, including the catalog search, so policy lookup is one tool among them. The model decides when a question is about policy. The server decides what the tool is allowed to return.
The contract is a pair of Zod schemas:
import { z } from 'zod';
const MAX_RESULTS = 3;
const QUERY_MAX_LENGTH = 200;
const boundedText = (maxLength: number) => z.string().trim().min(1).max(maxLength);
export const policyToolInput = z.strictObject({
query: boundedText(QUERY_MAX_LENGTH).describe(
'A policy question or keywords, for example "refund policy"',
),
});
const policySummary = z.strictObject({
docKey: boundedText(80),
titleVi: boundedText(200),
titleEn: boundedText(200),
bodyVi: boundedText(10_000),
bodyEn: boundedText(10_000),
});
export const policyToolOutput = z.union([
z.strictObject({
available: z.literal(true),
policies: z.array(policySummary).min(1).max(MAX_RESULTS),
}),
z.strictObject({
available: z.literal(false),
policies: z.array(policySummary).max(0),
reason: z.enum(['invalid_arguments', 'policy_unavailable', 'no_policy_match']),
}),
]);
export type PolicyToolOutput = z.infer<typeof policyToolOutput>;Each constraint closes a door:
strictObjectrejects unknown keys in both directions. The model can't pass arguments the tool doesn't define, and the tool can't leak extra columns such as scores or internal ids.- Success requires at least one document. "Succeeded, found nothing" can't be represented, so an empty search must come back as
no_policy_match. - Failure requires zero documents. "Failed, but here are a few anyway" can't be represented either.
- The query is capped at 200 characters. It's a search phrase, not a place to paste a document.
- The output is validated on the way out. If the tool ever produces something outside the contract, that's a bug and it throws. The model never sees a half-valid result.
The same discipline applies on the write side of an API: make the illegal states unrepresentable, then let the type do the arguing. Idempotency keys and optimistic locking does the same thing for retries.
How do you make citations that can't be faked?
The tempting approach is to tell the model "end with [1], [2] pointing at the sources". The trouble is that the model writes a citation the same way it writes everything else. It can produce a footnote that looks right and points at nothing, or at the wrong document. A citation the model writes is a claim, not evidence.
So the citation travels outside the prose. doc_key is unique, and it is the identity of the source. When the policy tool returns, the server emits a typed event on the response stream:
await emit({
type: 'tool_result',
kind: 'policy_faq',
payload: output.policies, // validated: doc keys, titles, bodies
});The client renders that payload as a card beside the model's answer. The reader sees the summary and, right next to it, the actual documents the model was given. If the summary drifts from the source, the drift is visible on screen. In exam terms, the examiner gets the page numbers from the library's checkout record, not from the student's handwriting.
Two more details follow the same idea. If the model provider fails after the tool has already returned documents, the server can answer from the tool result itself, using the first document's body cut to 1,000 characters, instead of improvising. And the stream event validates its payload with the same document schema as the tool output, so the client can't receive a shape the server didn't check.
To be clear about the limit: this proves which documents the model saw, not that every sentence of the answer follows from them. A sentence-level faithfulness check (a second model pass, or an entailment check against the cited text) isn't there.
How do you keep embeddings fresh when documents change?
An index is a copy, and copies go stale. The two corpora are fed differently.
Policy documents come from a seed script. Each entry carries a content version and a SHA-256 hash of its payload, and the upsert refuses to replace a row with a lower version.
Catalog items follow events from the service that owns them. The assistant's service treats each event as a signal, not as data. It records the event in an inbox keyed by event id (the same id with a different payload hash is an error, not a duplicate). Then it asks the owning service for the item's current state with a plain HTTP read, embeds it, and upserts it behind a version guard. Here is the rule as one statement:
INSERT INTO item_documents (item_id, search_text, source_version, embedding, embedding_model)
VALUES ($1, $2, $3, $4::vector, $5)
ON CONFLICT (item_id) DO UPDATE
SET search_text = EXCLUDED.search_text,
source_version = EXCLUDED.source_version,
embedding = EXCLUDED.embedding,
embedding_model = EXCLUDED.embedding_model
WHERE item_documents.source_version <= EXCLUDED.source_version;The repository does the comparison inside a transaction rather than in ON CONFLICT, but the rule is the same: a late, older event can't overwrite newer text. Where those events come from, and why none get lost on the way, is the transactional outbox.
A reconcile job runs every 15 minutes (cron 0 */15 * * * *). It retries up to 50 failed events. Then, if the document table is empty, it backfills the whole catalog. That covers items that existed before the consumer first ran, and reseeds that published no events. The same check runs at startup. When the table is already populated, the backfill is a no-op and spends no embedding quota.
There's a gap, and I'd rather name it. When an item is embedded during an embedding outage, its row is saved with a NULL vector. It stays keyword-searchable, which is the right fallback. But nothing re-embeds it until the item changes again: the reconcile job retries failed events, not missing vectors. The embedding_model column is stored on every row, which is exactly what you need when switching models (vectors from different models can't be compared), but no job reads it yet. The missing sweep is short:
SELECT item_id
FROM item_documents
WHERE embedding IS NULL OR embedding_model <> $1
ORDER BY item_id
LIMIT 50;Why not chunk the documents?
The standard advice is to split long documents into passages and embed each one. This system doesn't, and on purpose.
Each policy document is short and covers one topic. The whole document is the natural unit to cite: "see the refund policy" is a citation a person can check. Chunking also has a sharp edge. A boundary can separate a rule from its exception, so the retriever returns "refunds within the window" without the "except for…" in the next passage.
I'd start chunking once a document covers several topics, or grows long enough that one vector blurs its parts together. At that point the citation becomes a document key plus a section anchor, and retrieval should return a matching chunk's neighbours along with it.
What does this design cost?
- An external call per question. Every policy question embeds the query first. With a 10-second timeout and up to three attempts, a slow embedding API can burn most of a turn's time budget before the keyword fallback kicks in.
- Two languages per row. Every policy edit is two edits, and the search text mixes both languages in one bag of words.
- Postgres does the vector math. That's fine at this size, but the query as written won't scale without changing its shape.
What would I do differently?
Write the evaluation set first. I'd start with thirty to fifty real questions, each labelled with the expected doc_key or "none", including questions that should return nothing. Measure hits and false matches, then choose the gate and the weights. Without that, every threshold in this article is a guess.
Fuse ranks, not scores. Reciprocal Rank Fusion ignores the incompatible scales and only uses each document's position in the two lists. It was in the original design, and the weighted sum shipped instead. At minimum, ts_rank_cd's normalisation flag 32 (rank / (rank + 1)) would put the keyword score in 0–1.
Add the re-embed sweep for NULL vectors and model changes, and give the query embedding a tighter time budget than the document side.
Make the keyword side accent-insensitive with the unaccent extension in the text search configuration, so Vietnamese typed without diacritics still matches. The original design called for it; it didn't make the first cut.
Key takeaways
- A nearest neighbour always exists. Put the "is anything relevant?" gate in the
WHEREclause so an empty result is possible. - Hybrid search pairs vectors (for paraphrase) with full-text (for exact words). Let the model turn the question into a short keyword query.
- "Nothing matched" and "couldn't search" are different true sentences. Keep them as different typed outcomes.
- Make retrieval a strict tool whose success case can't be empty and whose failure case can't carry documents.
- Send citations as data (the document key and its text) beside the prose, never as footnotes the model writes.
- Freshness is a pipeline: an inbox, a version guard, a reconcile job, and a sweep for vectors that never got written.
- Untuned weights are guesses. Build the evaluation set before you tune anything.
A citation the model writes is a claim, and a document the server hands over is evidence. The same rule holds for a coding agent that reports "done": how to verify code an AI agent wrote.