Retrieval and RAG

Overview

Retrieval-augmented generation puts a search step in front of a language model. The user’s question is used to find the handful of passages most likely to contain the answer, those passages go into the prompt, and the model answers from them rather than from memory. It is the standard way to give a model access to private, recent or large bodies of text, and it is cheaper, more auditable and easier to update than fine-tuning for anything factual. This hub is for engineers building or debugging such a system.

The reason to understand the parts is that RAG failures are nearly always retrieval failures. When the answer is wrong, the first question is whether the right passage was in the prompt at all, and that question cannot be answered without instrumenting each stage separately.

Key ideas

Embeddings turn text into comparable vectors

An embedding model maps a passage to a vector of a few hundred to a few thousand numbers such that passages with similar meaning land close together, usually measured by cosine similarity. A bi-encoder embeds the query and the documents independently, which is what makes search over millions of documents possible: the document vectors are computed once and stored. The model’s maximum input length matters. all-MiniLM-L6-v2 (22 M parameters, 384 dimensions, truncates at 256 tokens, Apache-2.0, 2021) is the small default used in this site’s examples. bge-m3 (568 M, 1,024 dimensions, 8 k tokens, MIT, January 2024) and nomic-embed-text-v1.5 (137 M, 768 dimensions, 8 k tokens, Apache-2.0, February 2024) are common open choices for production. OpenAI’s text-embedding-3-large (3,072 dimensions, 8,191 tokens, January 2024, parameter count undisclosed) is a typical hosted option. Vectors from different models, or different versions of the same model, are not comparable, so changing the embedding model means re-indexing everything.

Chunking and indexing

Documents are split into chunks before embedding, because a vector for a whole report averages away the detail a question needs. Chunks of 200 to 500 tokens with a small overlap are the usual starting point, but splitting on structure (headings, paragraphs, code blocks, table rows) beats fixed character counts, and each chunk should carry metadata such as source, section title and date so that results can be filtered and cited. An index then makes nearest-neighbour search fast. A flat index compares against every vector and is exact, fine up to a few hundred thousand chunks. HNSW builds a graph of neighbours and answers in milliseconds at millions of vectors with recall above 95 percent at the cost of memory, roughly the vectors themselves (count times dimensions times 4 bytes) plus the graph. Product quantisation compresses vectors when memory is the limit.

Hybrid search and reranking

Dense vectors are good at paraphrase and poor at exact strings: part numbers, error codes, names and acronyms. BM25 keyword search is the reverse. Running both and merging the lists with reciprocal rank fusion is cheap and fixes a whole class of misses, which is why most production systems are hybrid. A reranker then re-scores the merged candidates with a cross-encoder, a model that reads the query and the passage together and is far more accurate than a bi-encoder but too slow to run over the whole corpus. Retrieve 50, rerank to 5 is the common shape. Query rewriting, where the model expands or restates the question before retrieval, helps with short or ambiguous queries and with multi-turn chat where the question depends on earlier turns.

Evaluate retrieval and generation separately

Build a test set of 50 to 100 real questions with the chunk or document that answers each. Measure retrieval with recall at k and mean reciprocal rank, and watch them as you change chunking, the embedding model or the fusion weights. Measure generation with faithfulness (does every claim trace to a provided chunk), answer relevance and, where there is a reference answer, correctness. RAGAS and similar libraries automate these with an LLM judge, which is useful for trend-tracking and not a substitute for reading a sample. Keep an eye on prompt length: Liu et al. found in 2023 that models use the middle of a long context poorly, so ten mediocre chunks can be worse than three good ones.

Where to start

Start with Embeddings and similarity search, which builds a nearest-neighbour search from scratch, then Using an LLM well for the prompting half and From notebook to service for serving it. The tiny RAG with local embeddings project assembles the pipeline in the diagram on a laptop with no API keys, and the RAG pipeline checklist is the pre-launch review.

Tools and platforms

Common mistakes

• Evaluating only the final answer. Without recall at k you cannot tell a retrieval failure from a prompting failure.
• Chunking by character count through tables, code and lists, so that the chunk with the answer starts mid-row.
• Embedding queries and documents with different models or different versions of one model, which makes every distance meaningless.
• Retrieving across tenants or permissions because the filter was applied after the top-k instead of inside the search.
• Raising top-k to fix a miss. More chunks dilute the prompt and push the right one into the middle of the context.
• Never re-indexing. Documents change, and a stale chunk with a confident citation is worse than no answer.

Going deeper

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, 2020.
• Malkov and Yashunin, the HNSW paper on approximate nearest neighbour search with navigable small world graphs, 2016.
• Nogueira and Cho, Passage Re-ranking with BERT, 2019, and Khattab and Zaharia, ColBERT, 2020.
• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, 2023.
• The sentence-transformers documentation and the FAISS wiki on choosing an index.