Failure labAdvanced125 min estimateOriginal publication

Diagnose RAG by Stage

An end-to-end answer score can tell you that the system is worse. It cannot tell you which team should change which component.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Evaluate RAG as a chain of conditional stages. First verify corpus coverage and freshness; then retrieval recall, reranking order, context sufficiency and contamination, answer correctness and faithfulness, citation entailment, latency, and cost. Keep stage outputs in the evaluation record so a regression remains attributable instead of collapsing into one opaque judge score.

Why this matters

Teams often tune the prompt when the answer was absent from the corpus, replace the embedding when an ACL filter removed the right document, or celebrate a judge score while citations point to irrelevant text. Stage isolation prevents expensive random walks.

You will be able to

Define an evaluation record that preserves every RAG stage.
Compute recall@k, reciprocal rank, and NDCG from labeled relevance.
Separate context sufficiency from answer faithfulness.
Design paired experiments that change one stage at a time.
Turn stage metrics into release gates and incident diagnosis.

Prerequisite contract

— Embeddings and vector retrieval
— Ranking metrics
RAG stage evaluation interview answer →

Your Vector Loop for this lab

01
Model
Represent RAG as evidence transformations with explicit inputs and outputs.
02
Derive
Derive retrieval metrics from relevance judgments.
03
Build
Create an evaluation record and pure metric functions.
04
Stress
Inject corpus, filter, ranking, context, and generation failures.
05
Operate
Gate releases by stage, slice, latency, cost, and safety.
06
Defend
Lead diagnosis from evidence instead of framework folklore.

Make every evidence transformation visible

01CorpusDoes an allowed, current source contain the answer?
02Candidate retrievalDid the retriever surface relevant evidence within k?
03RerankingDid the ranking stage promote the best evidence?
04Context assemblyDid packing preserve sufficient evidence without harmful distraction?
05GenerationIs the answer correct and supported by supplied context?
06CitationDo cited spans actually entail the associated claims?
07SystemDid the pipeline meet latency, cost, privacy, and reliability constraints?

Use ranking metrics that answer different questions

Recall@k = relevant retrieved within k / all relevant eligible items

Use recall when missing any relevant evidence is costly. It requires knowing the eligible relevant set.

MRR = mean(1 / rank of first relevant result)

Reciprocal rank rewards putting the first useful result early; it ignores additional relevant results after the first.

NDCG@k = DCG@k / ideal DCG@k

NDCG handles graded relevance and discounts useful items that appear late. It is suited to reranking when evidence quality is not binary.

Question	Primary metric	Blind spot
Did any answer evidence appear?	hit rate / recall@k	order within k
How early was the first useful item?	MRR	later relevant items
Is graded evidence ordered well?	NDCG@k	answer quality
Did the answer use supplied evidence?	faithfulness	whether evidence was correct
Did citations support claims?	citation entailment	uncited correct claims

Diagnose from the first failed boundary

RAG incident isolator

Select an observed production failure and inspect the evidence trace across candidate generation, ranking, context assembly, and generation. The isolator identifies the first unsupported boundary and proposes the next measurement—not a framework swap.

Observed failureThe answer is absent from every retrieved chunkThe right chunk is retrieved, but ranked below the context cutoffThe answer is in context, but the model contradicts it

First failed boundary: candidate generation

1
Candidate generationfail
No answer-bearing document appears in top k.
2
Rankingnot tested
No useful candidate exists to order.
3
Context assemblynot tested
The evidence cannot enter context.
4
Generationnot tested
Do not grade generation before retrieval.

Next measurement: Measure retrieval recall with answer-bearing document labels before changing the prompt.

Build metrics that fail loudly on bad labels

retrieval_eval.py

1from math import log2
2 
3def recall_at_k(relevances, k, total_relevant):
4    if total_relevant <= 0:
5        raise ValueError("total_relevant must be positive")
6    return sum(value > 0 for value in relevances[:k]) / total_relevant
7 
8def reciprocal_rank(relevances):
9    for rank, value in enumerate(relevances, start=1):
10        if value > 0: return 1 / rank
11    return 0.0
12 
13def dcg_at_k(relevances, k):
14    return sum((2 ** rel - 1) / log2(rank + 1) for rank, rel in enumerate(relevances[:k], start=1))
15 
16def ndcg_at_k(relevances, k):
17    ideal = dcg_at_k(sorted(relevances, reverse=True), k)
18    return 0.0 if ideal == 0 else dcg_at_k(relevances, k) / ideal
19 
20labels = [0, 2, 1, 0]
21print(f"{recall_at_k(labels, 3, 2):.2f}")
22print(f"{reciprocal_rank(labels):.2f}")
23print(f"{ndcg_at_k(labels, 3):.3f}")

Expected output

1.00
0.50
0.659

Verify: Run python -m unittest discover courses/ai-engineering/reference-impl/rag_eval.

The record around these functions matters more than the functions: query version, tenant and ACL context, corpus snapshot, embedding and index versions, retrieved IDs and scores, reranker version, final context spans, model and prompt versions, answer, citations, timings, token counts, and grader evidence.

Inject failures that end-to-end scores conceal

Injected fault	Retrieval	Generation	Correct diagnosis
answer document deleted from corpus	recall falls	unsupported or refuses	coverage/freshness
ACL removes correct document	tenant slice falls	may hallucinate	authorization/filtering
reranker reverses top two	candidate recall stable	answer quality falls	reranking
context truncates answer span	retrieved IDs look correct	incomplete answer	context assembly
model ignores supplied evidence	all retrieval metrics stable	faithfulness falls	generation/prompt
citation offsets drift	answer may be correct	citation entailment falls	citation mapping

Turn evaluation into a release and learning loop

01VersionFreeze the case set, corpus snapshot, pipeline configuration, graders, and rubrics.
02SliceReport by query intent, language, tenant, document type, freshness, difficulty, and safety class.
03CompareUse paired cases and confidence intervals; inspect large regressions even when the mean improves.
04GateSet stage-specific floors plus latency, cost, and safety constraints before running the candidate.
05ObserveSample production traces and convert representative failures into reviewed evaluation cases.
06RetireRemove stale or leaked cases and record why, instead of silently editing history.

Operate at three altitudes

Production lens

— Store evaluation traces with sensitive content controls and stable identifiers.
— Use paired comparisons and slice-level regression limits, not only aggregate means.
— Keep corpus coverage, retrieval, context, answer, citation, latency, and cost gates independent.
— Feed production failures back only after deduplication, labeling, and leakage review.

Staff lens

— Make ownership follow stage boundaries so a failure has a clear first responder.
— Define which user journeys can degrade, refuse, or fall back when evidence is insufficient.
— Require every architecture change to state which evaluation dimensions it expects to move and why.

Interview defense

How would you evaluate a production RAG system?

I would keep an end-to-end user outcome but decompose the pipeline into corpus coverage, candidate retrieval, reranking, context sufficiency, answer correctness and faithfulness, citation support, latency, cost, and safety. I would version stage outputs, use labeled retrieval metrics such as recall and NDCG, calibrate automated graders against humans, slice by real traffic segments, and gate releases on paired regressions.

Expect the interviewer to press on

— What if the corpus has no eligible answer?
— How do you distinguish retrieval failure from generation failure?
— How do you stop a golden set becoming stale?

Misconceptions to remove

“Answer accuracy is enough to evaluate RAG.”

It cannot reveal unsupported correct guesses, bad citations, retrieval waste, access-control failures, latency, or cost.

“Faithfulness proves the answer is true.”

Faithfulness only indicates support from supplied context; the context itself may be wrong or stale.

“A better embedding model improves the whole system.”

It may improve some retrieval slices while harming latency, index cost, multilingual behavior, or score calibration.

Check your model

1. Why can retrieval recall be undefined for a case?

If no eligible corpus item contains the answer, the denominator for relevant eligible items is zero; classify it as a corpus-coverage case instead.

2. What does stable recall with falling NDCG suggest?

Relevant candidates are still present, but their ordering or graded relevance worsened—usually a ranking issue.

Prove the mechanism

Create twelve cases spanning no-answer, ACL-filtered, stale, ambiguous, multi-hop, and adversarial queries. Label the first boundary that should fail safely.

Add a production constraint

Add bootstrap confidence intervals and paired regression reporting to the harness, then define a release gate that protects a long-tail slice.

Artifact: Stage-isolated RAG evaluation harness

courses/ai-engineering/reference-impl/rag_eval/retrieval_eval.py

Download reference implementation

Primary references and next links

References

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al.. Primary RAG paper combining parametric and non-parametric memory.
2. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Thakur et al.. Primary benchmark paper for retrieval across diverse tasks.
3. RAGChecker: A Fine-grained Framework for Diagnosing RAG
Ru et al.. Primary paper on fine-grained retriever and generator diagnosis.

Continue through the graph

RAG quality regression diagnosis →
Practice incident triage.
RAG quality loop →
Continue into staff-level release practice.
Production RAG system design →
Place the eval harness inside a complete architecture.

Glossary: recall@k · MRR · NDCG · faithfulness · citation entailment · golden set