RAG Architecture: Chunking, Embedding & Hybrid Retrieval

Why RAG Demands a Systems Mindset

Retrieval-augmented generation (RAG) is not a single model or a library call. It is a pipeline of interconnected subsystems: document ingestion, chunking, embedding, indexing, retrieval, reranking, and generation. Each stage introduces latency and quality trade-offs that compound. A naive RAG pipeline using a 7B generator with a 768-dim embedding model and a flat FAISS index might achieve 20 tokens/sec generation but spend 300ms per retrieval call, bottlenecking interactivity. On a single RTX 4090 (24GB VRAM, 1.0 TB/s bandwidth), you can run a 7B model at 4-bit (GGUF, ExLlamaV2) consuming ~5GB, leaving ~19GB for embedding models and vector store. If you use a 1.5B embedding model (e.g., intfloat/e5-mistral-7b-instruct quantized to 8-bit), that adds ~2GB. The remaining VRAM can hold a FAISS index of ~500k 768-dim vectors in float16 (~1.2GB). The key insight: RAG is a memory-bandwidth-bound system, not compute-bound. The embedding inference and retrieval steps compete for the same HBM bandwidth as the generator. On an H100 (80GB, 3.35 TB/s), you can run a 70B generator at 4-bit (~35GB) plus a full-precision 7B reranker (~14GB) and still have room for a 10M-vector index. But on an M3 Ultra (192GB unified memory, ~800 GB/s bandwidth), the bottleneck shifts to memory bandwidth for large-scale indexing. Understanding these hardware constraints is the first step to building a RAG system that doesn't feel like a prototype.

# Example: VRAM budget for local RAG on RTX 4090 (24GB)
# Generator: Llama-3-8B, 4-bit GGUF -> ~5.5 GB
# Embedding: bge-small-en-v1.5 (384-dim) -> ~0.5 GB (fp16)
# Reranker: BAAI/bge-reranker-v2-m3 (568M params, fp16) -> ~1.1 GB
# Vector store: FAISS index for 1M vectors (384-dim, float16) -> ~0.8 GB
# Total: ~7.9 GB, leaves headroom for KV cache (8k context ~2 GB)
# Remaining VRAM for OS/other: ~14 GB free, comfortable.

Chunking: The Overlooked Bottleneck

Chunking is the first and most impactful design decision in RAG. Fixed-length chunking (e.g., 512 tokens with 64-token overlap) is simple, deterministic, and easy to parallelize. However, it frequently splits sentences, code blocks, or logical sections, leading to incomplete contexts that confuse both the embedding model and the generator. Semantic chunking uses a sentence or paragraph boundary detector (e.g., spaCy, NLTK, or a dedicated segmentation model) to produce chunks that are coherent units. In benchmarks on the QMSum dataset, semantic chunking improves retrieval recall@5 by 8–12% over fixed-length chunking at the same average chunk size. But semantic chunking is slower: on a CPU with spaCy, you get ~500 tokens/sec; on an RTX 4090 with a transformer-based segmenter (e.g., tokenizer-free models), you can hit 5000 tokens/sec. For a 10GB document corpus, that's the difference between 6 hours and 36 minutes. Chunk overlap is critical: without overlap, a sentence that spans two chunks is lost. With 10% overlap, recall improves by 5–7% on the BEIR benchmark. Beyond 20% overlap, the index grows linearly and retrieval precision degrades due to duplicate vectors. For code repositories, chunk by function or class boundaries using a language parser (tree-sitter). For legal or medical documents, use section headers as natural breakpoints. The optimal chunk size depends on your generator's context window: for a 4k-context generator, chunks of 512 tokens with 64 overlap work well. For 128k-context models (e.g., Yi-34B-200K), you can use 2k-token chunks, but retrieval latency increases because the embedding model must process longer sequences. On a single A100, a 2k-token embedding takes ~2ms vs ~0.5ms for 512 tokens. That 4x increase matters at scale.

Embedding Models: Size vs. Specificity

Embedding models compress semantics into fixed-size vectors. The dominant paradigm is to use a bidirectional transformer (e.g., BERT, RoBERTa) with a mean pooling or CLS token output. Model sizes range from 22M params (all-MiniLM-L6-v2, 384-dim) to 7B params (intfloat/e5-mistral-7b-instruct, 4096-dim). Larger models generally capture more nuanced semantics, but the law of diminishing returns sets in quickly. On the MTEB benchmark, the 7B e5-mistral achieves an average score of 66.6, while the 110M bge-base-en-v1.5 scores 63.7. That 3-point gain costs 60x more compute and 30x more memory. For most local RAG deployments, a 384-dim model (bge-small, all-MiniLM) is sufficient for retrieval, provided you add a cross-encoder reranker. The reranker (e.g., BAAI/bge-reranker-v2-m3, 568M params) scores each retrieved chunk against the query jointly, achieving much higher precision than any bi-encoder alone. On the TREC-COVID dataset, a small bi-encoder + reranker outperforms a 7B bi-encoder by 4% in NDCG@10, while using 1/10th the VRAM. Quantization is essential for local deployment: embedding models can be quantized to 8-bit with negligible accuracy loss (less than 0.5% on MTEB). INT4 quantization drops 1–2% but saves 4x memory. On an RTX 5090 (32GB VRAM, 1.8 TB/s), you can run a 7B embedding model at 8-bit (~7GB) alongside a 70B generator at 4-bit (~35GB), but that leaves no room for a reranker or large index. The pragmatic choice: use a 384-dim model (0.5GB fp16) + a 1.1GB reranker + a 7B generator (4-bit, 5.5GB) for a total of ~7GB, leaving headroom for KV cache and index.

from sentence_transformers import SentenceTransformer

# Small, fast embedding model (384-dim)
model = SentenceTransformer('BAAI/bge-small-en-v1.5', device='cuda')
# Quantize to fp16 (default), no accuracy loss
chunks = ["RAG systems need careful chunking.", "Embedding models compress semantics."]
embeddings = model.encode(chunks, normalize_embeddings=True)  # (2, 384)
print(embeddings.shape)
# Output: (2, 384)

Indexing: FAISS, LanceDB, and the Memory Wall

Once chunks are embedded, you need an index for fast approximate nearest neighbor (ANN) search. FAISS (Facebook AI Similarity Search) is the gold standard for GPU-accelerated indexing. The two main index types for RAG are IVFPQ (inverted file with product quantization) and HNSW (hierarchical navigable small world). IVFPQ is more memory-efficient: a 1M vector index with 384-dim vectors in IVFPQ (M=64, nbits=8) uses ~300MB on disk and ~500MB in GPU memory. HNSW (e.g., HNSW32) uses ~2GB for the same dataset but offers 2–3x faster search and higher recall. On an H100, a GPU-accelerated IVFPQ index can achieve 10k queries/sec with recall@10 of 0.95. On an RTX 4090, expect 5k queries/sec. LanceDB is a newer alternative that uses a columnar format and supports hybrid search natively (dense + sparse via BM25). It is optimized for disk-based storage and incremental updates, making it ideal for dynamic document collections. For static corpora, FAISS is faster. For streaming data, LanceDB wins. Memory bandwidth is the bottleneck: loading a 384-dim vector from HBM takes ~0.6 microseconds on an H100 (3.35 TB/s). For a 1M vector index, brute-force search would take 0.6 seconds per query, unacceptable. ANN indices reduce this to 10–100 microseconds. The trade-off: higher recall requires more memory and more bandwidth. A common production configuration is IVF with 4096 centroids and 64 probes, giving recall@10 of 0.97 on standard benchmarks. For local RAG on a Mac M3 Ultra (unified memory), FAISS CPU mode with HNSW is the best option, achieving ~1k queries/sec for 1M vectors. Always normalize embeddings to unit length before indexing; cosine similarity is equivalent to inner product on normalized vectors, and FAISS's IndexIP works faster than IndexIDMap for normalized data.

import faiss
import numpy as np

# Assume embeddings: (N, 384), float32, normalized
d = 384
nlist = 4096  # number of centroids
quantizer = faiss.IndexFlatIP(d)  # inner product on normalized vectors
index = faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_INNER_PRODUCT)
index.train(embeddings)
index.add(embeddings)
index.nprobe = 64  # number of probes at search time

# Search
query = np.random.randn(1, d).astype(np.float32)
faiss.normalize_L2(query)
D, I = index.search(query, k=10)  # returns distances and indices

Hybrid Retrieval: Dense + Sparse Is Not Optional

Dense retrieval (embedding-based) excels at semantic similarity but fails on exact term matching. For example, a query for 'PCIe Gen 5 bandwidth 128 GB/s' might retrieve documents about 'NVLink bandwidth' because the embeddings capture the concept of bandwidth, not the specific term 'PCIe Gen 5'. Sparse retrieval (BM25) catches exact matches but misses synonyms and paraphrases. Hybrid retrieval combines both: retrieve top-k from each method, then merge and rerank. The standard approach is to use a weighted sum of dense and sparse scores, typically 0.3 * dense + 0.7 * sparse (or learn the weight via a small logistic regression on a validation set). In BEIR benchmarks, hybrid retrieval improves NDCG@10 by 5–15% over dense-only, and by 10–20% over sparse-only. For code retrieval, the gap is even larger: on the CodeSearchNet dataset, hybrid retrieval achieves 0.72 MRR vs 0.58 for dense-only. Implementation is straightforward: use FAISS for dense search and a BM25 index (e.g., rank_bm25 or Elasticsearch) for sparse. On a single RTX 4090, you can run both in parallel: dense search takes ~0.2ms, sparse search takes ~1ms (for a 1M document corpus). The merge step adds negligible latency. For a production system, consider learned sparse models like SPLADE (SPLADE-v3, 110M params) which produces sparse vectors that can be indexed in the same ANN index as dense vectors. SPLADE achieves near-hybrid quality with a single index, but the model is larger (110M vs 22M for BM25). On an H200 (141GB HBM3e, 4.8 TB/s), you can run SPLADE alongside a dense model and a 70B generator without breaking a sweat. On a consumer GPU, stick with BM25 + a small dense model.

Reranking: The Cost-Effective Quality Lever

Retrieval (whether dense, sparse, or hybrid) returns a shortlist of candidates. Reranking applies a more expensive but more accurate model to reorder these candidates. The standard reranker is a cross-encoder: a transformer that takes the query and a candidate chunk as a concatenated input (e.g., [CLS] query [SEP] chunk [SEP]) and outputs a relevance score. Cross-encoders are 10–100x slower than bi-encoders because they must process each query-chunk pair independently. However, they are also significantly more accurate: on the MS MARCO passage ranking task, a cross-encoder (e.g., BAAI/bge-reranker-v2-m3) achieves MRR@10 of 0.41 vs 0.34 for the best bi-encoder. The key is to limit the number of candidates passed to the reranker. Retrieving top-100 from the ANN index and reranking top-20 yields the same final recall@5 as retrieving top-20 and reranking all 20, but at 5x lower cost. On an RTX 4090, a 568M cross-encoder processes ~100 query-chunk pairs per second (batch size 1). For a user-facing application, you want reranking latency under 50ms. That means you can rerank at most 5–10 candidates per query. If your ANN index returns 100 candidates, you must either batch them (increasing latency) or use a smaller reranker. A practical configuration: retrieve top-50 from hybrid search, then rerank top-10 with a 110M cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) which runs at ~500 pairs/sec on an RTX 4090. The total retrieval + reranking latency stays under 20ms, leaving 80ms for generation (at 20 tokens/sec, that's ~1.6 tokens of generation time). For high-throughput systems, consider using a listwise reranker (e.g., RankGPT) but that requires an LLM call per reranking step, which is only viable on high-end hardware.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('BAAI/bge-reranker-v2-m3', device='cuda')
query = "How does PCIe Gen 5 bandwidth compare to Gen 4?"
candidates = ["PCIe Gen 5 offers 128 GB/s...", "NVLink bandwidth is 900 GB/s..."]
scores = reranker.predict([(query, c) for c in candidates])
best_idx = scores.argmax()
print(f"Best chunk: {candidates[best_idx]}")

Putting It Together: A Full Local RAG Pipeline on RTX 5090

Let's design a production-grade RAG pipeline for a 32GB RTX 5090. The generator is Llama-3-70B at 4-bit (GGUF, ~35GB), that alone would exceed 32GB, so we must use a smaller generator. Instead, use Llama-3-8B at 4-bit (~5.5GB) or Mistral-7B at 4-bit (~4.5GB). The embedding model: bge-small-en-v1.5 (384-dim, fp16, ~0.5GB). Reranker: cross-encoder/ms-marco-MiniLM-L-6-v2 (110M, fp16, ~0.4GB). Vector store: FAISS IVFPQ index for 5M chunks (~1.5GB). Sparse index: rank_bm25 in memory (~2GB for 5M documents). Total: ~10GB for retrieval components, leaving ~22GB for the generator and KV cache (8k context ~2GB). That's comfortable. The pipeline: user query -> embed query (0.2ms) -> dense search top-50 (0.3ms) -> BM25 search top-50 (1ms) -> merge and deduplicate top-50 -> rerank top-10 (10ms) -> pass top-3 to generator (generation: 50 tokens/sec for 8B model, ~200ms for 10 tokens). Total latency: ~212ms per query. Throughput: ~5 queries/sec. For higher throughput, you can batch queries: process 8 queries simultaneously, achieving ~40 queries/sec with the same hardware. On an H100 (80GB), you can scale to a 70B generator (4-bit, ~35GB) + 7B reranker (~7GB) + 10M index (~3GB) + BM25 (~4GB) = ~49GB, leaving ~31GB for KV cache (32k context ~8GB). Latency per query: ~300ms (dominated by generation). Throughput: ~20 queries/sec with batching. The key takeaway: local RAG is not only possible on consumer hardware, it's practical and performant. The bottleneck is almost always the generator, not the retrieval. Optimize generation first (quantization, FlashAttention, PagedAttention via vLLM), then tune retrieval.

Hardware-Specific Optimizations: MLX, TensorRT-LLM, and Triton

Different hardware platforms require different software stacks. On Apple Silicon (M3 Ultra), MLX is the native framework. MLX supports on-the-fly quantization of embedding models and efficient matrix multiplication on the unified memory architecture. A 384-dim embedding model runs at ~10k tokens/sec on M3 Ultra (192GB, ~800 GB/s). For generation, MLX-community models (e.g., mlx-community/Llama-3.2-3B-Instruct-4bit) achieve 60 tokens/sec. The unified memory means you can hold a 70B model at 4-bit (~35GB) plus a 7B reranker (~7GB) plus a 10M vector index (~3GB) in the same memory pool, with no PCIe transfers. The trade-off is that memory bandwidth is lower than an H100, so large-batch throughput suffers. On NVIDIA GPUs, TensorRT-LLM is the fastest inference engine. It compiles the model into optimized CUDA kernels, fusing operations and using FP8 quantization on H100/H200. For retrieval, Triton Inference Server can serve the embedding model and reranker concurrently with the generator, all on the same GPU. Triton's concurrent model execution allows overlapping retrieval and generation: while the generator processes one query, the next query's embedding and retrieval can run in parallel, hiding latency. On an H200, this can double throughput from 20 to 40 queries/sec. For AMD GPUs (MI300X, 192GB HBM3, 5.2 TB/s), ROCm with vLLM and FAISS (via hipCUB) is the path. The MI300X's large memory allows hosting a 70B generator at full precision (~140GB) plus a 7B reranker (~14GB) and a large index. However, software maturity lags behind NVIDIA: expect 80–90% of the performance for the same hardware specs. For Intel GPUs (Max 1550), use OpenVINO and IPEX. The ecosystem is improving but still niche for RAG workloads.

Benchmarking Your RAG Pipeline: Metrics That Matter

Do not rely on intuition. Benchmark your pipeline with domain-specific data. The three key metrics are retrieval recall (at k), generation faithfulness (e.g., via ROUGE-L or BERTScore), and end-to-end latency. For retrieval, use the BEIR benchmark suite but adapt it to your domain. For a code RAG system, use CodeSearchNet or a custom dataset of your repository. Measure recall@5, recall@10, and MRR. For generation, use the RAGAS framework (github.com/explodinggradients/ragas) which computes faithfulness, answer relevancy, and context precision. On a typical enterprise dataset, a well-tuned hybrid retrieval pipeline achieves recall@5 of 0.85–0.95. Adding a reranker pushes recall@5 to 0.92–0.98. Generation faithfulness (as measured by RAGAS) should be above 0.8 for a 7B generator with 4k context. If faithfulness drops below 0.7, your retrieval is likely returning irrelevant chunks. Latency budgets: for interactive applications, aim for <500ms end-to-end. For batch processing, throughput (queries/sec) is more important. Use a profiler (e.g., PyTorch profiler, NVIDIA Nsight) to identify bottlenecks. In our experience, the retrieval pipeline (embedding + search + reranking) accounts for 10–30% of total latency, generation for 70–90%. Optimize generation first: use FlashAttention-2, PagedAttention, and speculative decoding (Medusa, Eagle) to reduce generation latency by 2–3x. Then tune retrieval: reduce index probes, use smaller embedding models, and limit reranker candidates. The Pareto principle applies: 20% of optimization effort yields 80% of the latency reduction.

# Quick benchmark: measure retrieval latency
import time
import numpy as np

# Simulate 1000 queries
queries = ["What is HBM3e bandwidth?"] * 1000
start = time.time()
for q in queries:
    emb = embed_model.encode(q, normalize_embeddings=True)
    D, I = index.search(emb.reshape(1, -1), k=50)
    # BM25 search (omitted for brevity)
    # Rerank (omitted)
end = time.time()
print(f"Average retrieval latency: {(end-start)/len(queries)*1000:.2f} ms")

Pitfalls and Misconceptions

A common mistake is to treat RAG as a 'set and forget' system. Chunking strategies that work for Wikipedia fail for legal contracts or GitHub repos. Always validate on your data. Another pitfall is using the same embedding model for retrieval and reranking; this ignores the fundamental difference between bi-encoders (fast, lossy) and cross-encoders (slow, accurate). A third misconception is that larger embedding models always improve retrieval. In practice, a 384-dim model with a good reranker beats a 7B embedding model alone, at a fraction of the cost. Finally, many engineers underestimate the impact of chunk overlap. Too little overlap loses context; too much creates duplicate vectors that confuse the reranker and inflate latency. The optimal overlap is 10–20% of chunk size, but this should be tuned empirically. Also, beware of the 'curse of dimensionality' for very large embedding dimensions (e.g., 4096). While they capture more information, they require more memory and longer search times. For most local RAG deployments, 384 or 768 dimensions are sufficient.

Further reading

• FAISS: A Library for Efficient Similarity Search
• Sentence-Transformers: Multilingual Sentence Embeddings
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• RAGAS: Automated Evaluation of Retrieval Augmented Generation
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking