Retrieval-augmented generation looks trivial in a notebook. It collapses the moment real users, real documents and real latency budgets enter the picture.
Chunking Is the Hidden 80%
Naive fixed-size chunking destroys context. We use a hierarchical strategy: semantic splits on heading boundaries, fallback to recursive character splits, then a small overlap to preserve cross-chunk references.
Hybrid Retrieval Beats Pure Vector Search
Dense vectors miss exact-match terms. Sparse retrieval misses semantic intent. Combining both with reciprocal rank fusion consistently beats either alone by 15 to 25 percent.
Evaluation Is Non-Negotiable
Build a golden dataset before you write the retriever. Without evals, every prompt tweak is a coin flip.
Observability From Day One
Log every retrieval: query, retrieved chunks, scores, final answer, latency. When a user complains the answer is wrong, you need to trace it back in under a minute.
