RAG (Retrieval-Augmented Generation)

How RAG Works: The Two-Stage Architecture

In short

RAG works in two stages. First, a retriever converts the user's query into a vector embedding and searches a vector database for the most relevant document chunks. Second, a generator (LLM) receives the query plus the retrieved chunks as context and produces a grounded response.

Every RAG system has the same fundamental flow, regardless of framework or provider:

1. Indexing (offline). Documents are split into chunks (typically 200–1,000 tokens), each chunk is converted into a vector embedding using an embedding model (e.g., OpenAI text-embedding-3-large, Cohere embed-v3), and the vectors are stored in a vector database alongside the original text and metadata.
2. Retrieval (at query time). The user's query is embedded using the same model. The vector database performs a similarity search (cosine similarity or approximate nearest neighbor) and returns the top-k most relevant chunks.
3. Generation (at query time). The retrieved chunks are injected into the LLM's prompt as context. The model generates a response grounded in those chunks, often with inline citations back to the source documents.

This separation of retrieval and generation is what makes RAG powerful: the retriever handles knowledge lookup (fast, updatable, scalable), while the LLM handles reasoning and language generation.

Vector Databases: The RAG Knowledge Store

In short

A vector database stores document embeddings and enables fast similarity search — the core retrieval mechanism in RAG. Leading options include Pinecone (managed), Weaviate (open-source, hybrid search), Qdrant (open-source, Rust-based), Chroma (lightweight, Python-native), and Milvus (open-source, large-scale).

The vector database is where your knowledge lives in RAG. It stores embeddings (dense numerical representations of text) and supports fast approximate nearest neighbor (ANN) search.

Most production RAG systems also use metadata filtering alongside vector search. For example, filtering by document date, department, or access level before running the similarity search dramatically improves relevance for enterprise use cases.

RAG vs Fine-Tuning: When to Use Each

In short

RAG retrieves external knowledge at query time without modifying the model. Fine-tuning modifies the model's weights on a custom dataset. RAG is better when knowledge changes frequently, needs to be traceable, or the corpus is large. Fine-tuning is better when you need to change the model's behavior, tone, or reasoning patterns.

This is the most common architectural decision teams face. The two approaches solve different problems:

In practice, the two are not mutually exclusive. Many production systems use a fine-tuned model as the generator inside a RAG pipeline — getting both behavioral adaptation and grounded knowledge.

Enterprise RAG: Challenges and Best Practices

In short

Production RAG systems face challenges in chunking strategy, metadata filtering, evaluation (faithfulness, relevance, completeness), access control, and latency. The difference between a demo and a production RAG system is almost entirely in how well these challenges are handled.

A basic RAG demo takes an afternoon. A production-grade RAG system that enterprise teams trust takes engineering discipline across several dimensions:

• Chunking strategy. Fixed-size chunks are a starting point. Production systems use semantic chunking (split at paragraph or section boundaries), parent-child retrieval (retrieve the chunk, return the parent section), or recursive summarization for long documents.
• Metadata filtering. Pre-filter by date, department, document type, or access level before vector search. Without this, retrieval quality degrades rapidly as the corpus grows.
• Evaluation. Measure faithfulness (does the answer match the retrieved documents?), relevance (did retrieval return the right chunks?), and completeness (did the answer address the full question?). Frameworks like RAGAS and DeepEval automate these metrics.
• Access control. Enterprise RAG must enforce document-level permissions. A user should never receive an answer grounded in a document they don't have access to. This requires permission-aware indexing and retrieval.
• Hybrid search. Combine vector search (semantic similarity) with keyword search (BM25) for better recall. Weaviate and Elasticsearch support this natively; others require a custom fusion layer.

RAG as a Tool for AI Agents

In short

In agentic AI systems, RAG is implemented as a tool the agent can invoke — not a static pipeline. The agent decides when to retrieve, what query to send, and whether the retrieved context is sufficient. Anthropic's 'Building effective agents' guide (Dec 2024) recommends RAG as a core tool-use pattern.

RAG started as a standalone architecture pattern, but in modern AI systems it is increasingly used as one tool among many that an AI agent can invoke. The agent decides dynamically whether to retrieve documents, and can reformulate the retrieval query if initial results are poor.

Anthropic's December 2024 engineering guide on building effective agents explicitly recommends RAG as a core tool-use pattern. In this model, the agent's tool list might include: a RAG retrieval tool, a web search tool, a code execution tool, and domain-specific API tools. The agent reasons about which tool to call based on the user's query.

• Static RAG pipeline. User query goes directly to retrieval, then to generation. No decision-making. Good for simple Q&A use cases.
• Agentic RAG. An AI agent decides when to retrieve, what to retrieve, whether to re-rank or re-query, and when the answer is good enough to return. Better for complex, multi-step research tasks.

Most production RAG systems in 2026 are moving toward agentic RAG — where the retrieval step is a tool call within a broader agent loop, not a hardcoded pipeline stage.