Retrieval-Augmented Generation
What you’ll learn. Why and when to use retrieval-augmented generation; how embedding models and similarity search work; approximate nearest-neighbor indexes (HNSW, IVF, product quantization); chunking and indexing strategy; hybrid sparse-plus-dense retrieval with rank fusion; reranking; query rewriting; a GraphRAG overview; how to evaluate retrieval and generation separately; and the common failure modes. Code builds a small but complete pipeline.
Why RAG
Section titled “Why RAG”LLM weights hold broad but frozen, lossy knowledge. RAG gives the model relevant text at query time:
question -> retrieve top-k passages from your corpus -> put them in the prompt -> answer (with citations)Benefits: access to private and fresh data without retraining; attribution (cite sources, let users verify); easier updates and deletions (update the index, not the weights; important for access control and compliance); reduced hallucination when the answer is in the corpus. Compare with fine-tuning (Fine-Tuning in Practice), which is better for behavior, not facts.
With million-token context windows, “just put everything in the prompt” is tempting. It works for small corpora, but cost, latency, context rot (Prompting) and lack of access control still favour retrieval for larger or changing corpora. The two approaches combine: retrieve generously, then reason over a large window.
Embeddings
Section titled “Embeddings”A text embedding model maps text to a vector in (commonly to ) such that semantically similar texts are close. Mostly transformer encoders (or decoder LLMs with pooling) trained with contrastive learning: pull together (query, relevant passage) pairs, push apart in-batch negatives and mined hard negatives. The InfoNCE loss for query , positive and negatives :
Practical points:
- Similarity: cosine similarity (equivalently dot product on L2-normalized vectors) is standard.
- Asymmetry: queries are short questions, documents are long statements. Many models use instruction or prefix strings (“query:” / “passage:”) that you must apply as documented.
- Matryoshka embeddings are trained so that the first dimensions form a good lower-dimensional embedding, allowing cheaper storage and search by truncation.
- Choice: evaluate candidates on your data (the MTEB benchmark is a useful starting point but doesn’t predict domain performance). Dimension, max input length, multilingual support, license and cost vary.
- Changing the model means re-embedding everything. Version your index.
- Late-interaction models (ColBERT) keep one vector per token and score by max-similarity matching (“MaxSim”). Higher quality than single-vector retrieval at greater index size; multi-vector approaches are also used for document images.
import numpy as npfrom sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-small-en-v1.5") # any embedding modeldocs = ["Paris is the capital of France.", "The mitochondria is the powerhouse of the cell.", "HNSW is a graph-based ANN index."]D = model.encode(docs, normalize_embeddings=True) # (n, d) unit vectorsq = model.encode(["What index uses graphs for nearest neighbours?"], normalize_embeddings=True)scores = q @ D.T # cosine similarityprint(docs[int(scores.argmax())])Vector search
Section titled “Vector search”Exact search compares the query against all vectors: per query, fine up to roughly a million vectors with good hardware, too slow beyond. Approximate nearest neighbor (ANN) indexes trade a little recall for orders-of-magnitude speed.
Hierarchical Navigable Small World graphs (Malkov and Yashunin, 2016) build a multi-layer proximity graph. Upper layers are sparse “highways”; the bottom layer contains all points with edges to near neighbors. Search starts at the top, greedily moves to the neighbor closest to the query, descends a layer, and at the bottom does a best-first search with a candidate list of size efSearch. Complexity is roughly logarithmic in . Parameters: M (edges per node: more means higher recall and memory), efConstruction (build quality), efSearch (recall vs latency at query time). Strengths: excellent recall/latency, incremental inserts. Weaknesses: memory-hungry (vectors plus graph, all in RAM), deletes are awkward, build is slow.
IVF and product quantization
Section titled “IVF and product quantization”IVF (inverted file): cluster the vectors with k-means into nlist cells; at query time, scan only the nprobe cells whose centroids are nearest the query. Memory-light, and works well on disk, but recall depends on nprobe.
Product quantization (PQ) compresses vectors: split each into sub-vectors and quantize each with its own small codebook (e.g. 256 centroids = 1 byte), turning a 1024-float vector (4 KB) into bytes. Distances are computed from lookup tables. Combined as IVF-PQ it enables billion-scale search in modest memory; typically followed by exact re-scoring of the top candidates. Scalar quantization (int8) and binary quantization are simpler compressions; DiskANN/Vamana-style graph indexes target SSD-resident search.
Systems
Section titled “Systems”FAISS (library), pgvector (Postgres extension), and dedicated vector databases (Qdrant, Milvus, Weaviate, Pinecone, Elasticsearch/OpenSearch, and others). Considerations beyond raw speed: metadata filtering (filter by tenant, date, ACL while searching; naive post-filtering can return too few results), updates and deletes, hybrid search support, multi-tenancy, and operations. For many applications, pgvector or an existing search engine is sufficient.
import faiss, numpy as npd, n = 384, 100_000xb = np.random.randn(n, d).astype("float32"); faiss.normalize_L2(xb)index = faiss.IndexHNSWFlat(d, 32, faiss.METRIC_INNER_PRODUCT) # M=32index.hnsw.efConstruction = 200index.add(xb)index.hnsw.efSearch = 64xq = xb[:5]scores, ids = index.search(xq, 10) # top-10 by inner product (cosine, as normalized)Chunking
Section titled “Chunking”Documents must be split into passages short enough to be embedded precisely and fit several into the prompt. Chunking is among the most impactful, least glamorous choices.
- Fixed size with overlap (e.g. 300-800 tokens, 10-20% overlap): simple baseline.
- Structure-aware: split on headings, paragraphs, code blocks, table rows; keep a section together. Better than blind splitting for most documents.
- Semantic chunking: split where embedding similarity between adjacent sentences drops. Sometimes better; often not worth the cost.
- Hierarchical / parent-child: embed small chunks for precise matching, but return their larger parent (section) to the model for context.
- Contextualized chunks: prepend document title, section path, or an LLM-generated one-line context (“This chunk is from the 2023 Q2 report for ACME…”) before embedding and BM25 indexing. This addresses chunks that are ambiguous out of context and has been reported to cut retrieval failures substantially.
- Metadata: store source, section, date, author, permissions with each chunk for filtering and citation.
Trade-off: small chunks give precise embeddings but lose context; large chunks dilute the embedding and waste prompt budget. Tune on your eval set.
Hybrid retrieval
Section titled “Hybrid retrieval”Dense retrieval captures semantics but can miss exact strings: product codes, error messages, names, rare terms, numbers. BM25 (NLP Foundations) excels there but misses paraphrase. Hybrid search runs both and fuses ranks. The simple, robust fusion is Reciprocal Rank Fusion:
which needs no score calibration across systems. Learned sparse models (SPLADE) provide a bridge: sparse, inverted-index-compatible, but with learned term expansion.
def rrf(rankings: list[list[str]], k=60): scores = {} for ranking in rankings: for rank, doc_id in enumerate(ranking, start=1): scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank) return sorted(scores, key=scores.get, reverse=True)
fused = rrf([bm25_top50_ids, dense_top50_ids])[:50]Reranking
Section titled “Reranking”First-stage retrieval (bi-encoder) embeds query and document independently, which is fast but coarse. A cross-encoder reranker reads the query and a candidate together, attends across both, and outputs a relevance score. It is far more accurate but costs one forward pass per candidate, so it is applied only to the top 50-200 candidates, returning the best 5-10.
Typical funnel:
corpus (millions) -> hybrid retrieval top ~100 -> cross-encoder rerank -> top ~5-10 -> LLMRerankers are often the single highest-ROI addition to a RAG system. LLM-based rerankers (listwise prompting) are possible at higher cost.
Query-side improvements
Section titled “Query-side improvements”- Query rewriting / expansion: turn a conversational follow-up (“and what about 2023?”) into a self-contained query using chat history.
- Multi-query: generate several paraphrases, retrieve for each, fuse.
- HyDE: have the LLM write a hypothetical answer and embed that to retrieve documents similar to it.
- Decomposition: split multi-hop questions into sub-questions; retrieve per step.
- Routing: send queries to the right index or tool (documents vs SQL vs web search).
- Agentic RAG: the model decides when and what to retrieve, issuing multiple searches and reading results iteratively (Agents & Tool Use). Often better for complex questions, at higher latency and cost.
GraphRAG overview
Section titled “GraphRAG overview”Vector RAG retrieves isolated passages and struggles with questions that need global understanding (“What are the main themes across this corpus?”) or multi-hop relations across documents. GraphRAG (Microsoft, 2024, and many variants) uses an LLM to extract entities and relationships from chunks into a knowledge graph, detects communities of related entities, and pre-generates summaries of each community. Queries then use the graph: local queries traverse neighbors of relevant entities; global queries synthesize over community summaries. Costs: expensive indexing (many LLM calls), extraction errors, graph maintenance. Use when relational or whole-corpus questions matter; otherwise a well-tuned hybrid + rerank pipeline is simpler. Lighter variants (entity-linked chunks, document hierarchies like RAPTOR’s recursive summary trees) capture some benefits more cheaply.
Assembling the generation step
Section titled “Assembling the generation step”Put retrieved chunks in clearly delimited, numbered, source-labelled blocks; instruct the model to answer only from them, cite chunk IDs, and say so when the context lacks the answer.
def build_prompt(question, chunks): ctx = "\n\n".join(f"[{i+1}] (source: {c['source']})\n{c['text']}" for i, c in enumerate(chunks)) return f"""Answer the question using only the context. Cite sources like [1].If the context does not contain the answer, reply exactly: "I don't know based on the provided documents."
<context>{ctx}</context>
Question: {question}"""Retrieved text is untrusted data and a prompt-injection vector (Prompting). Apply document-level permissions at retrieval time so users never retrieve content they shouldn’t see.
Evaluation
Section titled “Evaluation”Evaluate retrieval and generation separately, otherwise you can’t tell which stage is failing (Evaluating LLMs).
Retrieval (needs labelled query-to-relevant-chunk data; build it from real queries, plus synthetic questions generated from chunks):
- Recall@k: fraction of relevant items in the top . The most important metric for the first stage, since the generator can’t use what wasn’t retrieved.
- MRR, nDCG@k, precision@k for ranking quality.
Generation:
- Faithfulness / groundedness: are claims supported by the retrieved context? (LLM-judge with claim-level checks, or NLI models.)
- Answer relevance and correctness versus a reference answer.
- Citation accuracy: do cited chunks support the sentence?
- Abstention quality: refuses when the answer isn’t present.
Frameworks (RAGAS, TruLens, DeepEval, and others) package these; validate their LLM-judge scores against human labels on a sample. Build the eval set early, include unanswerable and multi-hop questions, and track it in CI.
Failure modes
Section titled “Failure modes”- Retrieval miss (answer exists but isn’t retrieved): bad chunking, vocabulary mismatch, weak embedder, too-small , missing metadata filters. Diagnose with recall@k.
- Retrieved but ignored or buried: relevant chunk in the middle of a long context; too many distractors. Rerank, reduce , order by relevance.
- Wrong-but-plausible retrieval: near-duplicate, outdated or conflicting documents (old policy versions). Use dates, versions, deduplication and source authority.
- Hallucination despite context: the model blends parametric memory. Use stricter instructions, citations, and faithfulness checks.
- Lost context in chunks: pronouns and references without antecedents; fix with contextualized chunks and parent retrieval.
- Multi-hop and aggregation questions (counting, “compare across all”): top- retrieval can’t answer them; use agentic loops, SQL/structured tools, or GraphRAG.
- Tables, PDFs, scans and images: parsing quality is frequently the real bottleneck. Invest in layout-aware extraction, OCR, table-to-markdown, or vision-based document retrieval (Multimodality).
- Staleness and sync: index out of date or deleted content still retrievable.
- Security: ACL leaks, poisoned documents, injection via retrieved text.
- Cost/latency: embedding at scale, reranking latency, long contexts; cache embeddings and frequent queries.
Key takeaways
Section titled “Key takeaways”- RAG supplies fresh, private, attributable knowledge at query time; it complements rather than replaces fine-tuning and long context.
- Dense embeddings capture semantics; BM25 captures exact terms; combine them with RRF, then rerank with a cross-encoder.
- HNSW gives high recall with low latency in RAM; IVF-PQ compresses for huge corpora; filtering and updates matter in practice.
- Chunking, contextual enrichment and document parsing often matter more than the choice of vector database.
- Evaluate retrieval (recall@k) and generation (faithfulness, correctness) separately and with your own data.
- GraphRAG and agentic retrieval address global and multi-hop questions at higher cost.
Exercises
Section titled “Exercises”- Hands-on. Build a RAG pipeline over a documentation set with BM25 (
rank_bm25) and an embedding model; measure recall@5 on 30 questions you write for each of: BM25 only, dense only, RRF hybrid. - Conceptual. Why is a cross-encoder more accurate than a bi-encoder, and why can’t it be used for first-stage retrieval over millions of documents?
- Hands-on. Vary chunk size (128, 256, 512, 1024 tokens) and plot recall@5 and answer accuracy. Where is the optimum on your data?
- Conceptual. An HNSW index has recall 0.92 at
efSearch=32. What happens to latency and recall when you raise it, and how would you choose the value? - Conceptual. Design access control for a multi-tenant RAG system. Where do you enforce permissions?
Solutions (sketch)
- The cross-encoder sees query and passage tokens jointly, so attention can model fine-grained interactions; a bi-encoder compresses each to one vector independently. Cross-encoding requires a forward pass per (query, document) pair and nothing can be precomputed, so it is model calls per query.
- Both rise: more nodes are explored. Pick the smallest
efSearchmeeting your recall target (measured against exact search on a sample) within the latency SLO. - Enforce at retrieval via metadata filters (tenant/user/group IDs) inside the vector search, not after generation; also never place unauthorized content in the prompt, since the model cannot be trusted to withhold it.
Further reading
Section titled “Further reading”- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020).
- Karpukhin et al., “Dense Passage Retrieval” (2020); Khattab and Zaharia, “ColBERT” (2020); Formal et al., “SPLADE” (2021).
- Malkov and Yashunin, “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” (2016); Jegou, Douze, Schmid, “Product Quantization for Nearest Neighbor Search” (2011); Johnson, Douze, Jegou, “Billion-scale similarity search with GPUs” (FAISS, 2017).
- Cormack, Clarke, Buettcher, “Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods” (2009).
- Gao et al., “Precise Zero-Shot Dense Retrieval without Relevance Labels” (HyDE, 2022); Sarthi et al., “RAPTOR” (2024).
- Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization” (2024).
- Muennighoff et al., “MTEB: Massive Text Embedding Benchmark” (2022); Es et al., “RAGAS” (2023).
- Anthropic, “Introducing Contextual Retrieval” (2024), for contextualized chunks.