InterviewsVector
Arc 7
Failure labAdvanced125 min estimateOriginal publication

Diagnose RAG by Stage

An end-to-end answer score can tell you that the system is worse. It cannot tell you which team should change which component.

Authorship
InterviewsVector
Published / updated
2026-08-11 / 2026-09-20

The decision in one pass

Evaluate RAG as a chain of conditional stages. First verify corpus coverage and freshness; then retrieval recall, reranking order, context sufficiency and contamination, answer correctness and faithfulness, citation entailment, latency, and cost. Keep stage outputs in the evaluation record so a regression remains attributable instead of collapsing into one opaque judge score.

Why this matters

Teams often tune the prompt when the answer was absent from the corpus, replace the embedding when an ACL filter removed the right document, or celebrate a judge score while citations point to irrelevant text. Stage isolation prevents expensive random walks.

You will be able to

  • Define an evaluation record that preserves every RAG stage.
  • Compute recall@k, reciprocal rank, and NDCG from labeled relevance.
  • Separate context sufficiency from answer faithfulness.
  • Design paired experiments that change one stage at a time.
  • Turn stage metrics into release gates and incident diagnosis.

Your Vector Loop for this lab

  1. 01

    Model

    Represent RAG as evidence transformations with explicit inputs and outputs.

  2. 02

    Derive

    Derive retrieval metrics from relevance judgments.

  3. 03

    Build

    Create an evaluation record and pure metric functions.

  4. 04

    Stress

    Inject corpus, filter, ranking, context, and generation failures.

  5. 05

    Operate

    Gate releases by stage, slice, latency, cost, and safety.

  6. 06

    Defend

    Lead diagnosis from evidence instead of framework folklore.

Make every evidence transformation visible

  1. 01CorpusDoes an allowed, current source contain the answer?
  2. 02Candidate retrievalDid the retriever surface relevant evidence within k?
  3. 03RerankingDid the ranking stage promote the best evidence?
  4. 04Context assemblyDid packing preserve sufficient evidence without harmful distraction?
  5. 05GenerationIs the answer correct and supported by supplied context?
  6. 06CitationDo cited spans actually entail the associated claims?
  7. 07SystemDid the pipeline meet latency, cost, privacy, and reliability constraints?

Use ranking metrics that answer different questions

Recall@k = relevant retrieved within k / all relevant eligible items

Use recall when missing any relevant evidence is costly. It requires knowing the eligible relevant set.

MRR = mean(1 / rank of first relevant result)

Reciprocal rank rewards putting the first useful result early; it ignores additional relevant results after the first.

NDCG@k = DCG@k / ideal DCG@k

NDCG handles graded relevance and discounts useful items that appear late. It is suited to reranking when evidence quality is not binary.

QuestionPrimary metricBlind spot
Did any answer evidence appear?hit rate / recall@korder within k
How early was the first useful item?MRRlater relevant items
Is graded evidence ordered well?NDCG@kanswer quality
Did the answer use supplied evidence?faithfulnesswhether evidence was correct
Did citations support claims?citation entailmentuncited correct claims

Diagnose from the first failed boundary

RAG incident isolator

Select an observed production failure and inspect the evidence trace across candidate generation, ranking, context assembly, and generation. The isolator identifies the first unsupported boundary and proposes the next measurement—not a framework swap.

Diagnose RAG at the first attributable stage

Choose a frozen synthetic failure trace, inspect evidence from retrieval through generation, and commit to the first failed stage before revealing the diagnosis.

Observed failure

Each case holds corpus, query, index, cutoff, and model revision fixed. The records are authored teaching fixtures, not production telemetry.

Recorded stage evidence

Read in execution order. A later symptom does not move ownership past an earlier failed contract.

  1. 1
    Candidate generation

    Zero of three labelled answer-bearing documents appears in top 20.

  2. 2
    Ranking

    The ranker receives 20 candidates, none labelled answer-bearing.

  3. 3
    Context assembly

    The assembled context contains no supporting span.

  4. 4
    Generation

    The response abstains because no supplied span supports the fact.

Which stage owns the first failure?

Select a prediction, then check it against the recorded evidence.

Build metrics that fail loudly on bad labels

retrieval_eval.py
1def main() -> None:
2    labels = [0, 2, 1, 0]
3    print(f"{recall_at_k(labels, 3, 2):.2f}")
4    print(f"{reciprocal_rank(labels):.2f}")
5    print(f"{ndcg_at_k(labels, 3):.3f}")

Expected output

1.00
0.50
0.659

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/rag_eval.

The downloadable implementation validates concrete bounded inputs, rejects booleans and non-finite labels, and keeps the supplied ranking unchanged before running the metric core shown above. The record around these functions matters more than the functions: query version, tenant and ACL context, corpus snapshot, embedding and index versions, retrieved IDs and scores, reranker version, final context spans, model and prompt versions, answer, citations, timings, token counts, and grader evidence.

Inject failures that end-to-end scores conceal

Injected faultRetrievalGenerationCorrect diagnosis
answer document deleted from corpusrecall fallsunsupported or refusescoverage/freshness
ACL removes correct documenttenant slice fallsmay hallucinateauthorization/filtering
reranker reverses top twocandidate recall stableanswer quality fallsreranking
context truncates answer spanretrieved IDs look correctincomplete answercontext assembly
model ignores supplied evidenceall retrieval metrics stablefaithfulness fallsgeneration/prompt
citation offsets driftanswer may be correctcitation entailment fallscitation mapping

Turn evaluation into a release and learning loop

  1. 01VersionFreeze the case set, corpus snapshot, pipeline configuration, graders, and rubrics.
  2. 02SliceReport by query intent, language, tenant, document type, freshness, difficulty, and safety class.
  3. 03CompareUse paired cases and confidence intervals; inspect large regressions even when the mean improves.
  4. 04GateSet stage-specific floors plus latency, cost, and safety constraints before running the candidate.
  5. 05ObserveSample production traces and convert representative failures into reviewed evaluation cases.
  6. 06RetireRemove stale or leaked cases and record why, instead of silently editing history.

Operate at three altitudes

Production lens

  • — Store evaluation traces with sensitive content controls and stable identifiers.
  • — Use paired comparisons and slice-level regression limits, not only aggregate means.
  • — Keep corpus coverage, retrieval, context, answer, citation, latency, and cost gates independent.
  • — Feed production failures back only after deduplication, labeling, and leakage review.

Staff lens

  • — Make ownership follow stage boundaries so a failure has a clear first responder.
  • — Define which user journeys can degrade, refuse, or fall back when evidence is insufficient.
  • — Require every architecture change to state which evaluation dimensions it expects to move and why.

Interview defense

How would you evaluate a production RAG system?

I would keep an end-to-end user outcome but decompose the pipeline into corpus coverage, candidate retrieval, reranking, context sufficiency, answer correctness and faithfulness, citation support, latency, cost, and safety. I would version stage outputs, use labeled retrieval metrics such as recall and NDCG, calibrate automated graders against humans, slice by real traffic segments, and gate releases on paired regressions.

Expect the interviewer to press on

  • — What if the corpus has no eligible answer?
  • — How do you distinguish retrieval failure from generation failure?
  • — How do you stop a golden set becoming stale?

Misconceptions to remove

“Answer accuracy is enough to evaluate RAG.”

It cannot reveal unsupported correct guesses, bad citations, retrieval waste, access-control failures, latency, or cost.

“Faithfulness proves the answer is true.”

Faithfulness only indicates support from supplied context; the context itself may be wrong or stale.

“A better embedding model improves the whole system.”

It may improve some retrieval slices while harming latency, index cost, multilingual behavior, or score calibration.

Check your model

1. Why can retrieval recall be undefined for a case?

If no eligible corpus item contains the answer, the denominator for relevant eligible items is zero; classify it as a corpus-coverage case instead.

2. What does stable recall with falling NDCG suggest?

Relevant candidates are still present, but their ordering or graded relevance worsened—usually a ranking issue.

Prove the mechanism

Create twelve cases spanning no-answer, ACL-filtered, stale, ambiguous, multi-hop, and adversarial queries. Label the first boundary that should fail safely.

Add a production constraint

Add bootstrap confidence intervals and paired regression reporting to the harness, then define a release gate that protects a long-tail slice.

Artifact: Stage-isolated RAG evaluation harness

courses/ai-engineering/reference-impl/rag_eval/retrieval_eval.py

Download reference implementation

Primary references and next links

References

  1. 1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Lewis et al.. Primary RAG paper combining parametric and non-parametric memory.

  2. 2. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

    Thakur et al.. Primary benchmark paper for retrieval across diverse tasks.

  3. 3. RAGChecker: A Fine-grained Framework for Diagnosing RAG

    Ru et al.. Primary paper on fine-grained retriever and generator diagnosis.

Continue through the graph

Glossary: recall@k · MRR · NDCG · faithfulness · citation entailment · golden set