Glossary/ Inference

RAG (Retrieval-Augmented Generation)

RAG augments LLM prompts with relevant documents retrieved from an external knowledge base, grounding responses in factual data to reduce hallucination and enable dynamic knowledge.

RAG (Retrieval-Augmented Generation) is a system design pattern that combines a retriever (e.g., vector database + embedding model) with a generative LLM. Before generating a response, the system first searches a corpus of documents (e.g., company wikis, product manuals, proprietary research) for chunks semantically similar to the user query. These retrieved chunks are then inserted into the LLM's prompt as context. This ensures the generated answer is grounded in specific source material, vastly reducing hallucination compared to relying solely on the model's parametric memory. RAG is particularly valuable for knowledge-intensive tasks like technical support, legal document analysis, and medical Q&A, where information changes rapidly or is proprietary. The typical RAG pipeline: (1) segment documents into chunks (256-512 tokens), (2) embed each chunk using a fine-tuned embedding model, (3) store embeddings in a vector database (e.g., Pinecone, Milvus, FAISS), (4) at query time, embed the user question, (5) retrieve top-k (5-20) most similar chunks, (6) assemble a prompt with system instruction, retrieved context, question, and (7) feed to the LLM. Advanced RAG incorporates re-ranking, query rewriting, and hybrid search (dense + BM25). The quality of the embedding model (e.g., BGE, E5, Voyage) directly impacts retrieval precision. AI builders must be careful about context window limits-too many retrieved chunks can truncate important information. RAG is not a silver bullet: if no relevant documents exist, the LLM may still guess. It also adds latency (retrieval + re-ranking) and requires ongoing document maintenance. Hybrid approaches (RAG + fine-tuning) often yield best results.

Common misconception

RAG does not update the LLM's parameters-it changes the input prompt dynamically; the model itself remains unchanged between queries.

Canonical reference

https://arxiv.org/abs/2005.11401

Related terms

Concepts you'll usually encounter alongside this one.

Embedding

Runtime & Tooling

An embedding is a dense vector representation of text that captures semantic meaning, enabling similarity search, clustering, and retrieval for tasks like RAG and semantic search.

Vector Database

Runtime & Tooling

A specialized database that stores and indexes high-dimensional vector embeddings for efficient similarity search in AI and LLM retrieval-augmented generation (RAG) pipelines.

Context Window

Architecture

The context window defines the maximum number of input tokens a transformer model can process at once, determining the length of conversation history or document it can reason over.