The first version of the assistant answered every question. That was the problem. Ask it something the documentation did not cover and it produced a confident, plausible, entirely invented answer — which is worse than no assistant at all, because a wrong answer about a refund policy costs more than a blank page.
Retrieval before generation
A language model asked a bare question answers from whatever it absorbed in training. Retrieval-augmented generation changes the order: search your own corpus first, put the passages you found into the prompt, and ask the model to answer from those.
The model stops being the source of facts and becomes the thing that phrases them. That is a much smaller job, and a much more reliable one.
Chunking decides the ceiling
Everything downstream is limited by how the corpus was cut up, and this is where most of the quality is won or lost.
Chunks that are too large bury the relevant sentence among paragraphs of noise, and the embedding averages out into something that matches everything weakly. Too small and the sentence loses the context that made it meaningful — "it must be returned within 14 days" is useless without knowing what it is.
Two things worked better than tuning the size:
Split on structure, not on character count. Headings, list items and table rows are boundaries the author already chose. Cutting every 500 characters severs sentences and merges unrelated sections.
Carry the heading into the chunk. Prefixing each chunk with its document title and section heading costs a few tokens and makes an otherwise ambiguous paragraph retrievable. It is the cheapest quality improvement available.
A similarity floor, and the willingness to say no
Vector search always returns its top k. It has no notion of "nothing here is relevant" — ask about the weather and it will still hand back the five least-unrelated passages about refunds.
So the floor has to be explicit:
const hits = await search(embedding, { limit: 8 })
const relevant = hits.filter((hit) => hit.score >= SIMILARITY_FLOOR)
if (relevant.length === 0) return { answer: NO_ANSWER, sources: [] }
Returning "I do not have that in the documentation" is a correct answer. Getting a system to admit that is most of what separates a useful assistant from a liability, and the floor is worth tuning on real questions rather than guessed ones.
Cite, and make the citation checkable
Every answer carries the passages it came from, and the interface shows them. This is not decoration — it does three things at once: the reader can verify without trusting the model, an unsupported claim becomes visible instead of silent, and the citations tell you which documents are actually load-bearing so you know what to keep current.
If the model produces a sentence that no cited passage supports, that is a bug you can now see.
What it cost
Embeddings are cheap to compute and easy to forget to invalidate. The lesson learned the hard way: the embedding is a cache of the document, and it must be invalidated exactly like one. Store a hash of the chunk with its vector and re-embed on change, or the assistant will confidently quote a policy you rewrote last month.
Latency is retrieval plus generation, and retrieval is the part you control. Keeping k small and the floor honest costs less than it sounds — fewer, better passages produce better answers than more, worse ones.
What I would keep
Of everything in that pipeline, two pieces earn their place unconditionally: the similarity floor, because it is what allows the system to be wrong less often than it is right, and the citations, because they are what let anyone else check. The rest is tuning.
