← AI Terminology

RAG - Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) is an architecture that enhances LLM responses by first retrieving relevant documents or data from an external knowledge base and including them in the context — grounding generation in retrieved facts rather than relying solely on the model's parametric knowledge.

It is the dominant architecture for building LLM applications over private or up-to-date knowledge.
Why It Matters in AI
LLMs have knowledge cutoffs and limited context — they cannot access company documents, recent news, or private databases. RAG solves this: embed documents into a vector store, retrieve the most relevant ones at query time, and include them in the LLM's prompt. This reduces hallucination (model can cite retrieved text), enables knowledge updates without fine-tuning, and allows LLMs to work with proprietary enterprise data. It is the most widely deployed LLM architecture in enterprise AI.
Key Points
Aspect Description
GraphRAG Microsoft: build knowledge graph from documents → graph-traversal retrieval for complex QA
Reranking Cohere Rerank, cross-encoder — reorder retrieved chunks by relevance after initial retrieval
Hybrid search Combine vector (semantic) + BM25 (keyword) retrieval — often better than either alone
Indexing phase Documents → chunked → embedded → stored in vector store (Chroma, Pinecone, Weaviate, pgvector)
Retrieval phase Query embedded → nearest-neighbour search → top-k relevant chunks retrieved
Generation phase Retrieved chunks + query → LLM prompt → grounded, cited response
Simple Analogy
An open-book exam: instead of answering purely from memory (parametric LLM), the student can look up relevant pages in the textbook (retrieval) and include that information in their answer (generation). The answer is grounded in retrieved evidence — more accurate and verifiable than pure memorisation.
Common Usage Examples
  • VectorStoreIndex.from_documents(documents).as_query_engine() — LlamaIndex RAG pipeline
  • RetrievalQA.from_chain_type(llm=ChatOpenAI(), retriever=chroma.as_retriever(k=5)) — LangChain RAG
  • pgvector: SELECT content FROM chunks ORDER BY embedding <=> $1 LIMIT 5 — Postgres vector search
  • from sentence_transformers import SentenceTransformer; embedder = SentenceTransformer("all-mpnet-base-v2")
  • Hybrid: EnsembleRetriever([bm25_retriever, vector_retriever], weights=[0.4, 0.6]) — LangChain
Summary
In short: RAG retrieves relevant documents from an external knowledge base and includes them in the LLM's context — grounding generation in retrieved facts, reducing hallucination, and enabling LLMs to work with private, proprietary, or up-to-date knowledge.