Design a Production RAG System in 2026

A basic retrieval-augmented generation demo can be built in an afternoon: split a few files, create embeddings, find nearest neighbors, and pass them to a language model. That design fails in production because it has no answer for deleted documents, permission changes, exact identifiers, stale caches, index migrations, or the question every incident commander asks: “Which stage became worse?”

The interview answer must cover data paths, quality, failures, and capacity—not just the vector database.

Quick Answer

A production RAG system separates versioned ingestion from online answering. It performs authorization-filtered BM25 and dense retrieval in parallel, fuses and reranks candidates, assembles a cited context, and generates or abstains. Evaluate each stage independently, and key caches by tenant, permissions, corpus snapshot, embedding, reranker, prompt, and generator versions.

TL;DR

• Start with quality, freshness, authorization, latency, availability, and cost.
• Ingest through an idempotent pipeline with canonical document IDs, content hashes, ACL metadata, delete propagation, and versioned chunks.
• Combine BM25 lexical search with dense retrieval. Use reciprocal rank fusion (RRF) when score scales differ and there is not enough training data.
• Rerank a bounded set, then deduplicate, diversify, and expand parent context.
• Apply access control before search and treat retrieved content as untrusted.
• Evaluate retrieval, ranking, context, answer, citation, and abstention quality separately. A single end-to-end score cannot diagnose regressions.
• Cache by authorization, corpus snapshot, and data/model versions.
• Roll out indexes blue-green with shadow traffic and rollback.

What the Highest-Ranking RAG Guides Miss

Current search results explain the components, but rarely close the operational loop. The missing pieces are usually:

The missing pieces are usually:

• ACL and deletion propagation across lexical and vector indexes;
• why retrieval filters must run before nearest-neighbor selection;
• how BM25 and vector results are fused without pretending their scores share a scale;
• cache keys that encode permissions, freshness, and model versions;
• zero-downtime embedding and index migrations;
• stage-level diagnosis across parsing, retrieval, ranking, and generation;
• how candidate counts translate into reranker throughput and GPU demand;
• tests for citations, abstention, poisoning, injection, and isolation; and
• graceful degradation when search, reranking, or generation fails.

Start the Interview with Requirements

State assumptions first. These numbers are a worked example.

Functional requirements

1. Ingest PDFs, HTML, Markdown, tickets, and structured records.
2. Answer natural-language questions with passage-level citations.
3. Respect tenant, user, group, region, and document permissions.
4. Update changed content within five minutes and remove revoked content faster.
5. Support exact lookups, semantic questions, and multi-document synthesis.
6. Abstain when authorized evidence is missing or contradictory.

Non-functional requirements

Clarify read patterns, cross-document synthesis, privacy, and whether outputs drive high-impact decisions. Risk changes abstention and review requirements.

End-to-End RAG Architecture

The architecture has three responsibilities:

• Data plane: ingestion and online query execution.
• Control plane: connectors, schemas, model and index versions, policies, experiments, rollouts, and kill switches.
• Quality plane: offline evaluation, online metrics, human review, and feedback that becomes new test data.

Keep ingestion off the request path and evaluation writes out of production.

Step-by-step Solution

1. Build an Idempotent Ingestion Pipeline

Each source record needs a stable identity. A useful chunk identity is:

chunkId = hash(tenantId, sourceId, sourceVersion, parserVersion, chunkerVersion, chunkOrdinal)

Hashes detect content changes; versions order events and make transformations reproducible. Keep canonical documents outside serving indexes so either index can be rebuilt without recrawling sources.

Parse, normalize, and preserve structure

Extract structure, page numbers, language, provenance, and ACLs. Remove navigation noise without destroying sections. Preserve OCR coordinates and confidence.

Chunk for the question, not for a fixed token count

Start with structure-aware chunks and tune on real questions.

Retrieve small child passages, then expand a bounded parent after reranking. Do not embed an entire long document as one vector.

Make updates and deletes first-class

Use at-least-once events and idempotent writes. Deletes create tombstones, remove both index entries, and invalidate caches. Advance the corpus snapshot after stores agree; dead-letter malformed inputs rather than retaining old entries silently.

2. Version Every Artifact That Can Change Meaning

Different embedding models or dimensions cannot share a similarity space. Prompt and permission changes also invalidate otherwise fresh answers.

For migration, dual-write, backfill a green index, reconcile ACL counts, shadow queries, then canary reads. Move the alias after quality and latency pass; retain the blue index through rollback and cache expiry.

3. Authenticate, Authorize, and Route the Query

Resolve identity and entitlements from verified credentials, then create an index-side tenant and ACL prefilter.

Route exact lookups to authorized APIs or SQL and navigation to search. Use RAG for unstructured synthesis. If rewriting follow-ups, trace the original query and test that exact terms survive.

4. Retrieve Lexically and Semantically in Parallel

BM25 handles exact strings, rare terms, and identifiers. Dense retrieval handles paraphrases and concepts. Production traffic contains both.

The official Elastic hybrid search guidance recommends combining full-text and vector retrieval, while the Azure AI Search ranking documentation explains why BM25 and vector scores have different ranges. Do not add their raw scores.

Reciprocal rank fusion

RRF combines positions rather than incompatible score magnitudes:

RRF(d) = Σ 1 / (k + rankᵣ(d))

Each retriever contributes by rank. The constant k controls how strongly top positions dominate. Start near 60, then tune. The official Elasticsearch RRF reference documents the same rank-based formula.

Test 50–200 candidates per branch, union by canonical chunk ID, and trace each rank. Tune with Recall@k and reranker capacity.

RRF versus weighted score fusion

5. Rerank a Bounded Candidate Set

First-stage retrieval favors recall and speed; reranking favors precision. A cross-encoder jointly reads the question and passage. The Sentence Transformers retrieve-and-rerank documentation describes the standard pattern: retrieve a broad candidate set, then apply the slower cross-encoder only to those candidates.

Rerank in batches, enforce a strict deadline, and record the model version. After reranking:

1. remove exact and near-duplicate passages;
2. cap passages per document or source to preserve diversity;
3. expand child chunks to a parent section when needed;
4. resolve contradictory versions using effective dates and source authority; and
5. stop when the context token budget or marginal relevance threshold is reached.

If reranking fails, use fused order only when evaluated as safe; otherwise abstain. Never wait on an unbounded GPU queue.

6. Assemble Evidence Before Generating

Give passages immutable citation IDs, versions, and access-safe URLs. Delimit retrieved text as untrusted evidence, never as instructions.

Context assembly maximizes relevant, independent evidence under a token budget. Irrelevant passages raise cost and can lower quality.

The generation contract should require:

• an answer supported only by the supplied passages;
• citation IDs attached to factual claims;
• a clear statement when evidence is insufficient or conflicting;
• no fabricated source, link, policy, or number; and
• structured output that a citation validator can parse.

Validate every cited ID and quoted span. Run stronger claim checks synchronously for high-impact uses. Return passages so readers can inspect evidence.

7. Design Caching Around Correctness

Caching is a set of layers with different invalidation rules.

Hash the effective tenant, principal, groups, and policy version into an ACL fingerprint without raw secrets. Large group sets can use a short-lived server-side authorization snapshot ID.

Semantic answer caching is risky: nearby questions can differ by date, jurisdiction, account, or negation. Restrict it to tested intents, store the source versions and policy, and reauthorize every hit. The Redis semantic caching documentation shows similarity-based response caching, but similarity alone is not a correctness boundary.

Coalesce identical misses and jitter TTLs. Do not cache aborted streams, denials, transient errors, or personalized secrets in a shared boundary.

8. Evaluate the Pipeline Stage by Stage

Build a versioned golden set from consented queries. Include answerable, unanswerable, exact-ID, temporal, multilingual, permission, and adversarial cases. Store expected sources and grading notes.

Recall@k measures whether top k contains relevant evidence. MRR rewards the first relevant rank; nDCG supports graded relevance. Slice by query class and language.

Frameworks such as Ragas can compute context and response metrics, but LLM judges are measurements, not ground truth. The Ragas documentation distinguishes LLM-based metrics, which can be non-deterministic, from deterministic metrics. Calibrate judges against human annotations, measure agreement, pin judge versions, and inspect disagreements.

Run the frozen suite in CI and change one major variable at a time. Online signals such as citation clicks and reformulations are useful but noisy.

9. Add Observability and Graceful Degradation

Trace authentication, each retrieval branch, fusion, reranking, assembly, generation, citation validation, and caches. Record IDs and scores; redact content by policy.

Tag corpus, index, model and prompt versions, candidate counts, cache decisions, tokens, tenant tier, and degraded-mode reason.

Use deadlines, breakers, bounded retries, bulkheads, and load shedding. Retry only transient, idempotent operations.

10. Secure the RAG Supply Chain

RAG does not remove prompt injection. The OWASP LLM01 prompt-injection guidance explicitly includes malicious instructions planted in documents retrieved by a RAG application.

Apply these controls:

• authorize before retrieval and again before opening a source link;
• scan and quarantine new content, while preserving provenance and signer data;
• treat retrieved text, metadata, HTML, images, and links as untrusted;
• keep tool execution behind deterministic authorization and confirmation;
• redact secrets and personal data before logs, evaluation exports, and caches;
• encrypt data, and rate-limit ingestion and queries;
• audit document changes, policy decisions, index versions, and answer evidence;
• red-team direct and indirect injection, data exfiltration, Unicode obfuscation, malicious citations, and cross-tenant probes; and
• define retention for sources, embeddings, logs, and evaluations.

Use the NIST Generative AI Profile for lifecycle governance, alongside a concrete threat model.

Minimal Reproducible Hybrid Retrieval Example

This standard-library program demonstrates BM25, a deterministic dense stand-in, RRF, reranking, authorization prefiltering, and a versioned cache.

The hashed vector only makes the demo reproducible. Replace it and rerank with versioned model adapters and durable stores that preserve the same invariants.

"""rag_demo.py: runnable mechanics for tenant-safe hybrid retrieval."""
 
from __future__ import annotations
 
import hashlib
import json
import math
import re
from collections import Counter
from dataclasses import dataclass
from typing import Iterable
 
 
TOKEN_RE = re.compile(r"[a-z0-9]+")
SYNONYMS = {
    "credential": "key",
    "credentials": "key",
    "rollover": "rotate",
    "zero": "without",
    "downtime": "downtime",
}
 
 
@dataclass(frozen=True)
class Document:
    id: str
    tenant: str
    title: str
    text: str
 
 
@dataclass(frozen=True)
class Hit:
    document: Document
    lexical_score: float
    dense_score: float
    fused_score: float
    rerank_score: float
 
 
DOCUMENTS = [
    Document(
        id="a-key-rotation",
        tenant="tenant-a",
        title="Production key rotation",
        text=(
            "Rotate production API keys without downtime by creating a second "
            "active key, deploying clients, verifying new-key usage, and only "
            "then revoking the old key."
        ),
    ),
    Document(
        id="a-password-reset",
        tenant="tenant-a",
        title="Password reset",
        text="Reset a user password after identity verification and revoke sessions.",
    ),
    Document(
        id="a-release",
        tenant="tenant-a",
        title="Safe deployment",
        text="Use canary releases, health checks, and rollback thresholds for services.",
    ),
    Document(
        id="b-private-key",
        tenant="tenant-b",
        title="Tenant B secret procedure",
        text="The emergency production API key is stored in Tenant B's private vault.",
    ),
]
 
 
def tokens(text: str) -> list[str]:
    """Normalize text; a real system versions its analyzer and synonym set."""
    raw = TOKEN_RE.findall(text.lower())
    return [SYNONYMS.get(token, token) for token in raw]
 
 
def bm25_scores(query: str, documents: list[Document]) -> dict[str, float]:
    """Compute BM25 scores over the already-authorized candidate corpus."""
    if not documents:
        return {}
    query_terms = tokens(query)
    document_terms = [tokens(f"{doc.title} {doc.text}") for doc in documents]
    average_length = sum(map(len, document_terms)) / len(document_terms)
    document_frequency = Counter(
        term for terms in document_terms for term in set(terms)
    )
    scores: dict[str, float] = {}
    k1, b = 1.2, 0.75
    for doc, terms in zip(documents, document_terms):
        frequencies = Counter(terms)
        score = 0.0
        for term in query_terms:
            frequency = frequencies[term]
            if frequency == 0:
                continue
            df = document_frequency[term]
            inverse_document_frequency = math.log(
                1 + (len(documents) - df + 0.5) / (df + 0.5)
            )
            denominator = frequency + k1 * (
                1 - b + b * len(terms) / average_length
            )
            score += inverse_document_frequency * frequency * (k1 + 1) / denominator
        scores[doc.id] = score
    return scores
 
 
def hashed_vector(text: str, dimensions: int = 64) -> list[float]:
    """Create a deterministic demo vector; not a production embedding model."""
    vector = [0.0] * dimensions
    for token in tokens(text):
        digest = hashlib.sha256(token.encode("utf-8")).digest()
        index = int.from_bytes(digest[:4], "big") % dimensions
        sign = 1.0 if digest[4] % 2 == 0 else -1.0
        vector[index] += sign
    norm = math.sqrt(sum(value * value for value in vector)) or 1.0
    return [value / norm for value in vector]
 
 
def dense_scores(query: str, documents: list[Document]) -> dict[str, float]:
    query_vector = hashed_vector(query)
    return {
        doc.id: sum(
            left * right
            for left, right in zip(
                query_vector,
                hashed_vector(f"{doc.title} {doc.text}"),
            )
        )
        for doc in documents
    }
 
 
def ranked_ids(scores: dict[str, float]) -> list[str]:
    return [item[0] for item in sorted(scores.items(), key=lambda item: item[1], reverse=True)]
 
 
def reciprocal_rank_fusion(rankings: Iterable[list[str]], k: int = 60) -> dict[str, float]:
    fused: dict[str, float] = {}
    for ranking in rankings:
        for rank, document_id in enumerate(ranking, start=1):
            fused[document_id] = fused.get(document_id, 0.0) + 1.0 / (k + rank)
    return fused
 
 
def rerank_score(query: str, document: Document, fused_score: float) -> float:
    """Deterministic relevance stand-in; a production adapter calls a reranker."""
    query_terms = set(tokens(query))
    document_terms = set(tokens(f"{document.title} {document.text}"))
    coverage = len(query_terms & document_terms) / max(len(query_terms), 1)
    exact_phrase = 1.0 if "without downtime" in document.text.lower() else 0.0
    return 4.0 * coverage + exact_phrase + fused_score
 
 
class HybridRetriever:
    def __init__(self, documents: list[Document]) -> None:
        self.documents = documents
        self.cache: dict[str, list[Hit]] = {}
 
    def retrieve(
        self,
        *,
        tenant: str,
        acl_fingerprint: str,
        query: str,
        index_snapshot: str,
        limit: int = 2,
    ) -> tuple[list[Hit], bool]:
        cache_material = {
            "tenant": tenant,
            "acl": acl_fingerprint,
            "query": " ".join(tokens(query)),
            "snapshot": index_snapshot,
            "embedding": "demo-hash-v1",
            "reranker": "demo-reranker-v1",
        }
        cache_key = hashlib.sha256(
            json.dumps(cache_material, sort_keys=True).encode("utf-8")
        ).hexdigest()
        if cache_key in self.cache:
            return self.cache[cache_key], True
 
        # Security boundary: filter before either top-k operation.
        authorized = [doc for doc in self.documents if doc.tenant == tenant]
        lexical = bm25_scores(query, authorized)
        dense = dense_scores(query, authorized)
        fused = reciprocal_rank_fusion([ranked_ids(lexical), ranked_ids(dense)])
 
        hits = [
            Hit(
                document=doc,
                lexical_score=lexical[doc.id],
                dense_score=dense[doc.id],
                fused_score=fused[doc.id],
                rerank_score=rerank_score(query, doc, fused[doc.id]),
            )
            for doc in authorized
        ]
        hits.sort(key=lambda hit: hit.rerank_score, reverse=True)
        self.cache[cache_key] = hits[:limit]
        return hits[:limit], False
 
 
def main() -> None:
    retriever = HybridRetriever(DOCUMENTS)
    request = {
        "tenant": "tenant-a",
        "acl_fingerprint": "groups:engineering;policy:7",
        "query": "How do I rotate a production API key without downtime?",
        "index_snapshot": "corpus-2026-08-03T12:00Z",
    }
 
    first, first_cached = retriever.retrieve(**request)
    second, second_cached = retriever.retrieve(**request)
 
    print(f"first_cache_hit={first_cached}")
    print(f"second_cache_hit={second_cached}")
    for rank, hit in enumerate(first, start=1):
        print(
            f"{rank}. {hit.document.id} tenant={hit.document.tenant} "
            f"rerank={hit.rerank_score:.3f}"
        )
    assert first == second
    assert all(hit.document.tenant == "tenant-a" for hit in first)
    assert all(hit.document.id != "b-private-key" for hit in first)
 
 
if __name__ == "__main__":
    main()

Save the file as rag_demo.py and run:

python3 rag_demo.py

Expected output:

first_cache_hit=False
second_cache_hit=True
1. a-key-rotation tenant=tenant-a rerank=3.833
2. a-password-reset tenant=tenant-a rerank=0.432

The early authorized filter and cache material are the critical lines. Removing the filter leaks candidates; omitting tenant, acl, or snapshot enables unsafe reuse. Apply the filter to both retrievers.

Capacity and Latency Planning

By Little's Law, 500 QPS at 2.5 seconds means about 1,250 requests in flight. Size each downstream pool separately.

A sample p95 budget might be:

Reranking 100 passages at 500 QPS creates 50,000 pairs per second before batching. Benchmark actual lengths, batches, models, and hardware before choosing CPU or GPU.

Raw vector bytes are chunks × dimensions × bytes per component. One hundred million 768-dimensional float32 vectors need about 307 GB decimal before graph, metadata, replicas, and headroom. Validate quantization against recall.

Common Causes

Most RAG failures are pipeline contract failures, not a generic hallucination.

Symptoms

Look for these patterns:

• exact ticket numbers fail while broad conceptual questions work;
• the right passage appears at rank 40 but the answer cites rank 1 noise;
• a deleted policy continues to appear only on repeated questions;
• one user sees a title or answer from a document they cannot open;
• answer quality falls after a parser release while generation metrics look stable;
• latency spikes only for long questions or large passages;
• quality improves offline while production escalation worsens;
• answers change across replicas due to version skew;
• two locales hit one cached response; or
• p95 is acceptable while p99 grows with a reranker or generation queue.

Root Cause

Fix the earliest failing stage. Generation cannot recover evidence never retrieved.

How to Reproduce and Debug a Bad Answer

Capture a redacted replay bundle with query, authorization and corpus snapshots, all versions, branch candidates, ranks, context, citations, caches, and timings.

Then follow this decision path.

Replay exact versions against the original and current snapshots. Inspect examples as well as aggregates; averages can hide a broken query class.

Troubleshooting Matrix

Verification Steps

Use explicit release gates:

1. Data: reconcile sources, chunks, embeddings, index records, ACLs, and deletes.
2. Retrieval quality: meet Recall@k by query class, language, corpus age, and tenant size. Compare lexical-only, dense-only, fused, and reranked variants.
3. Answers: review correctness, grounding, citations, and abstention.
4. Isolation: prove that tenant A identities cannot retrieve, cache-hit, cite, preview, or open tenant B content. Include group removal during an active session.
5. Freshness: edit and delete canary documents; measure every store and cache until the change is visible.
6. Resilience: inject timeouts, rate limits, corrupt data, and queue overload.
7. Performance: load test realistic lengths and report tail latency, queues, throughput, tokens, and cost.
8. Migration: shadow the green index, validate coverage and ACL parity, canary traffic, exercise rollback, and let old cache entries expire before deletion.
9. Security: red-team direct and retrieved prompt injection, poisoned sources, data exfiltration, malicious links, PII in logs, and cache-key confusion.
10. Operations: confirm alerts, runbooks, owners, kill switches, and audit retention.

Prevention

Prevent repeat failures with engineering controls:

• Make canonical IDs, content hashes, ACLs, provenance, and versions mandatory schema fields.
• Reject serving records that lack an authorization predicate or source version.
• Run ingestion reconciliation and deletion canaries continuously.
• Add every material incident to a frozen regression set.
• Gate changes on stage-level quality, latency, cost, safety, and isolation—not one average score.
• Centralize cache-key construction so a caller cannot omit tenant or model fields.
• Use blue-green indexes and reversible aliases for schema or embedding changes.
• Cap candidate windows, context tokens, queue depth, retries, and generated output.
• Shadow rewriting, semantic caching, and learned fusion.
• Review low-confidence and high-impact answers with humans; preserve the evidence and version bundle needed for an audit.

Deployment and Platform Notes

The Docker resource-constraint documentation explains CPU and memory controls; without them a container can consume host resources. For GPU containers, follow the official Docker GPU-access guide. Windows GPU support is documented through Docker Desktop's WSL 2 backend.

Alternative Architectures

Add graph, multi-query, or agentic retrieval only for measured failures.

A 45-Minute RAG System-Design Interview Plan

Connect every choice to a requirement: hybrid for mixed query types, prefilters for privacy, RRF for incompatible scores, and stage metrics for diagnosis.

Key Takeaways

• RAG is versioned retrieval with probabilistic generation, not a vector feature.
• Hybrid retrieval protects both exact-match and semantic use cases. Fuse ranks before reranking; do not add raw BM25 and vector scores.
• Authorization belongs inside retrieval, and authorization context belongs inside cache keys.
• Evaluation must identify the earliest failing stage. Retrieval recall sets an upper bound on what reranking and generation can recover.
• Caching is safe only when its key and invalidation rules model every dependency.
• Blue-green indexes and replay bundles make failures reversible and diagnosable.

Suggested Internal Links

• How vector indexes trade recall for latency
• BM25 explained for system-design interviews
• Choosing an embedding model for production search
• Designing multi-tenant authorization filters
• LLM evaluation metrics and release gates
• Semantic caching without data leakage
• Prompt-injection defenses for retrieved content
• Blue-green index migrations and rollback

FAQs

What is the best architecture for a production RAG system?

Separate versioned ingestion from online queries. Authenticate, prefilter lexical and dense retrieval, fuse and rerank candidates, assemble cited context, then generate or abstain and validate citations.

Why is hybrid search better than vector search alone for RAG?

Dense search finds concepts; lexical search finds exact codes and names. Fuse their ranks with RRF, then tune candidates and fusion on labeled queries.

How many chunks should a RAG system retrieve and rerank?

Retrieve enough candidates to meet Recall@k, then rerank the smallest window that preserves recall within latency. Test tens to low hundreds and measure on target models, lengths, batches, and hardware.

Which RAG evaluation metrics matter most?

Measure Recall@k, MRR or nDCG, context precision/recall, and human-calibrated answer, citation, and abstention quality. Track tail latency, errors, queues, tokens, and cost separately.

What should be included in a RAG cache key?

Include request, tenant, authorization fingerprint, locale, corpus snapshot, search settings, and every relevant model, prompt, policy, and decoding version. No key may omit a correctness or access dependency.

How should a RAG system handle document deletion?

Write an ordered tombstone, remove both index entries, invalidate derived caches, and advance the serving snapshot after reconciliation. Verify deletion canaries against the revocation SLO.

Does RAG prevent hallucination or prompt injection?

No. Models can misread evidence or follow instructions inside it. Require grounded answers and valid citations, separate instructions from data, enforce permissions outside the model, and red-team indirect injection.

When is a cross-encoder reranker worth the cost?

Use one when retrieval recall is good, top-rank precision is weak, and evaluation shows enough gain. Bound and batch the workload; skip it when search is already precise or latency exceeds budget.

How do you migrate to a new embedding model?

Build a new index, dual-write, backfill canonical documents, reconcile ACL counts, shadow queries, canary traffic, then move the read alias. Keep the old index through rollback and cache expiry.

Should RAG use a vector database, a search engine, or PostgreSQL?

Choose by benchmark. Hybrid engines integrate BM25 and vectors; vector databases specialize in ANN; PostgreSQL can simplify moderate workloads and metadata joins. Compare recall, filtering, updates, tail latency, operations, and cost.