How to Implement RAG in Production: 7 Architecture Patterns with Code

Step-by-Step Guide

1. 1

   Pattern 1: Naive RAG

   The simplest RAG pattern: chunk documents, embed chunks, store in a vector database, and retrieve top-k chunks for each query. This works for simple Q&A but struggles with complex questions requiring multi-hop reasoning.

   python Copy

   def naive_rag(query, vector_db, llm, k=5):
       query_emb = embed(query)
       chunks = vector_db.search(query_emb, top_k=k)
       context = "\n\n".join(chunks)
       return llm.generate(f"Context: {context}\nQuestion: {query}")

2. 2

   Pattern 2: Advanced RAG with Re-ranking

   Retrieve a larger candidate set (top-20), then re-rank with a cross-encoder model like Cohere Rerank or BGE-Reranker. Re-ranking is slower but significantly improves relevance. This is the most common production pattern.

   💡

   Tip: Re-ranking typically improves answer accuracy by 15-25% over naive retrieval. The cost is latency — add 100-300ms per query.
3. 3

   Pattern 3: Multi-Query RAG

   Generate multiple reformulations of the user query using the LLM, retrieve for each, and merge results. This captures different phrasings of the same intent and improves recall. Use reciprocal rank fusion to merge the multiple result lists.

   python Copy

   def multi_query_rag(query, llm, vector_db):
       queries = llm.generate(f"Rewrite this question 3 ways: {query}")
       queries = [query] + parse(queries)
       all_results = [vector_db.search(embed(q), top_k=10) for q in queries]
       return reciprocal_rank_fusion(all_results, top_k=5)

4. 4

   Pattern 4: Graph-Based RAG

   Build a knowledge graph from your documents, then use graph traversal alongside vector search. This handles multi-hop questions like "What companies did the CEO of SpaceX found before SpaceX?" by following entity relationships.

   ⚠️

   Warning: Graph-based RAG requires significant upfront investment in entity extraction and relationship modeling. Only worth it for complex, multi-hop question domains.
5. 5

   Pattern 5: Agentic RAG

   Give the LLM tools to search, retrieve, and decide when it has enough information. The agent can make multiple retrieval calls, filter results, and decide to search a different database. This is the most flexible but also the most expensive and hardest to control.

   

6. 6

   Pattern 6: Hybrid RAG (Keyword + Semantic)

   Combine BM25 keyword search with dense vector search. Keyword search excels at exact matches (names, codes, IDs), while semantic search handles conceptual similarity. Use reciprocal rank fusion or a learned fusion model to combine results.

   python Copy

   def hybrid_rag(query, bm25_index, vector_db, alpha=0.5):
       bm25_results = bm25_index.search(query, top_k=20)
       vec_results = vector_db.search(embed(query), top_k=20)
       fused = reciprocal_rank_fusion([bm25_results, vec_results])
       return fused[:5]

7. 7

   Pattern 7: Multi-Modal RAG

   Extend RAG to include images, tables, and charts. Use multi-modal embeddings (CLIP, GPT-4V) to embed both text and images in the same vector space. This enables querying across documents that contain visual information like diagrams and screenshots.

   💡

   Tip: Multi-modal RAG is particularly valuable for technical documentation where critical information is in diagrams, not text.
8. 8

   Optimize Chunking Strategy

   Chunking dramatically affects RAG quality. Fixed-size chunks (512 tokens) are simple but can split important context. Semantic chunking (splitting on topic boundaries) is better but slower. For code, chunk by function/class. For markdown, chunk by header section.

   ⚠️

   Warning: Chunk size is a critical hyperparameter. Too small = insufficient context. Too large = diluted relevance signal. 256-512 tokens is a good starting point.
9. 9

   Implement Evaluation Pipeline

   Build automated RAG evaluation using RAGAS or TruLens. Track: Context Precision (are retrieved chunks relevant?), Context Recall (did we miss important context?), Faithfulness (is the answer grounded in retrieved context?), Answer Relevance (does the answer address the question?).

   python Copy

   from ragas import evaluate
   from ragas.metrics import context_precision, faithfulness
   
   scores = evaluate(
       dataset=rag_dataset,
       metrics=[context_precision, faithfulness, answer_relevance]
   )
   print(f"Faithfulness: {scores['faithfulness']:.2f}")

10. 10

    Deploy with Caching and Streaming

    Cache embeddings for repeated queries. Cache LLM responses for identical query+context pairs. Stream responses to the user for perceived latency reduction. Use semantic caching (cache responses for similar queries, not just identical ones) for additional speedup.

    💡

    Tip: Semantic caching with a similarity threshold of 0.95 can reduce LLM calls by 30-50% in production.
11. 11

    Monitor and Iterate

    In production, track: retrieval latency, LLM latency, cache hit rate, user feedback (thumbs up/down), and faithfulness scores. Set up alerts for faithfulness drops — this usually indicates the knowledge base has changed or the chunking strategy needs updating.

12. 12

    Handle Edge Cases

    Plan for: queries with no relevant context (return "I don't know" instead of hallucinating), queries that need real-time data (add a web search tool), queries that require multiple documents (implement multi-hop retrieval), and queries in different languages (use multilingual embeddings).

13. 13

    Scale to Production

    For production scale: use a managed vector database (Pinecone, Weaviate Cloud, pgvector), implement rate limiting, add authentication, use async/await for concurrent requests, and deploy behind a CDN. Target p99 latency under 2 seconds for the full RAG pipeline.