Systems labAdvanced110 min estimateOriginal publication

Reranking and Context Assembly

A reranker can improve the ordering of candidates it sees. It cannot recover a missing passage, choose an unlimited context, or prove that a generator will use the evidence correctly.

Authorship: InterviewsVector
Published / updated: 2026-09-20 / 2026-09-20
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Executable illustrative contracts have focused tests and recorded primary sources. No named human reviewer, benchmark result, or security certification is claimed.

The decision in one pass

Treat retrieval, reranking, and context assembly as three contracts. Candidate generation optimizes broad coverage under a cheap latency budget. A cross-encoder or other expensive reranker jointly evaluates the query and each supplied candidate, improving local ordering but never rescuing evidence outside that set. Context assembly then applies a separately versioned token budget, duplicate policy, source cap, ordering rule, provenance frame, and model/tokenizer contract. Bind rerank scores to the exact query and candidate content in generator order; reject stale or incomplete score vectors; pack deterministically; and evaluate candidate recall, reranking quality, context utilization, answer grounding, and latency at their own stages.

Why this matters

Teams often celebrate a better reranker metric while answer quality remains flat because the relevant chunk never entered the candidate set, the packing policy dropped it, or the model ignored it in a long prompt. Separating the stages preserves attribution. It also prevents an apparently harmless tokenizer, source-diversity, or context-order change from silently altering the evidence a generator receives.

You will be able to

Distinguish cheap candidate generation from joint query-passage reranking and final context packing.
Explain why a reranker can reorder but cannot repair candidate recall.
Bind scores to exact query, candidate order, chunk content, model, and evaluation revisions.
Apply deterministic token, chunk, duplicate, and source constraints after reranking.
Test score, tokenizer, context-position, latency, and constructor-bypass failure modes.
Measure retrieval, reranking, assembly, and generation with stage-appropriate evidence.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Draw candidate generation, reranking, packing, generation, and evaluation as separate components with exact inputs, budgets, and owners.
02
Derive
Derive the per-candidate cross-encoder scoring cost and the constrained selection problem imposed by token, chunk, source, and ordering policies.
03
Build
Bind supplied scores to an immutable candidate set and emit a deterministic context plan with explicit omissions.
04
Stress
Remove the relevant candidate, reorder score bindings, duplicate text, corrupt token counts, exceed source caps, and vary evidence position.
05
Operate
Observe candidate recall, reranker ranking metrics, packing omissions, context utilization, generation grounding, latency, and cost separately.
06
Defend
Defend the assembled context as an evaluated plan under one model/tokenizer policy, not as proof of relevance, grounding, or optimal order.

Give each stage one job

Stage	Primary job	What it cannot establish
candidate generation	find a broad, bounded set cheaply	best final order or answer support
reranking	judge query-candidate relevance more deeply	relevance of candidates it never received
context assembly	select and frame evidence within serving constraints	that the generator will use evidence faithfully
generation	produce an answer under the supplied context	retrieval recall or source authority

A bi-encoder can precompute document representations and search a large collection efficiently because query and document are encoded separately. A cross-encoder reads the query and candidate together, allowing richer token interactions at a cost paid for every scored pair. This makes the common architecture a funnel: retrieve tens or hundreds, rerank fewer, pack fewer still.

Bind every score to the pair that produced it

Ci = Retrieve(q, K); si = fcross(q, ci); π = sort(C, s)

The reranker score is conditional on the exact query, candidate text, truncation policy, model revision, and scoring head. Sorting is meaningful only after preserving that identity.

A vector of scores without ordered candidate content is not replayable evidence. The artifact records a content digest for every chunk, the generator rank, a query digest, the ordered candidate-set digest, and an equally long score sequence. Replacing one chunk, reordering candidate IDs, or reusing scores from another query changes the evidence identity and is rejected.

Reranker outputs are supplied declarations in this teaching artifact. Production evidence should link an authenticated inference record to exact model weights, tokenizer, query, candidate bytes, truncation behavior, and serving configuration. A content hash detects substitution; it does not prove that a model actually emitted the score.

Turn an ordering into a bounded context plan

Select P ⊆ π subject to Σi∈P tokens(ci) ≤ B, |P| ≤ M, and count(source(ci)) ≤ S

The fixture uses a deterministic greedy walk through reranked candidates with a total token budget B, maximum chunks M, and per-source cap S. This is one inspectable policy, not an optimal solver.

Token counts must come from the tokenizer revision used by the serving model; character counts are not a safe substitute.
Near-duplicate chunks waste context and can make one source appear independently corroborated. Detect duplication before packing and retain source identity.
A source cap can create diversity but may discard multiple necessary passages from one document. Evaluate multi-hop and single-source tasks separately.
Framing should preserve chunk, document, revision, source, and citation identity while clearly marking retrieved text as untrusted data.

Audit scores, then pack under an explicit contract

reranking_context_audit.py

1def assemble_context(
2    contract: AssemblyContract,
3    candidates: CandidateSet,
4    evidence: RerankEvidence,
5) -> AssemblyReport:
6    """Audit supplied rerank scores, then pack under an explicit local policy."""
7    contract = validate_record(contract, AssemblyContract)
8    candidates = validate_record(candidates, CandidateSet)
9    evidence = validate_record(evidence, RerankEvidence)

Expected output

example=illustrative_only
status=ASSEMBLED_FOR_EVALUATION
selected=chunk-b,chunk-c,chunk-d
tokens=12;sources=3
claim=ILLUSTRATIVE_PLAN_NOT_RERANKING_BENCHMARK

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/reranking_context

The fixture contains four invented candidates and supplied scores. It selects three chunks totaling twelve declared tokens from three sources. The public function reconstructs every nested record, verifies query and scope identities, requires an exact candidate-order binding, sorts by score with stable tie keys, and records why each omitted chunk hit a token, source, or chunk limit.

The script does not retrieve, run a cross-encoder, tokenize text, verify provenance, choose an optimal subset, or call a language model. ASSEMBLED_FOR_EVALUATION is therefore a plan awaiting evaluation, not evidence of reranking quality or answer grounding.

Measure the first stage that could explain the failure

01Candidate recallAt the reranker input depth, ask whether any judged supporting passage was available. Segment results by corpus, query, ACL, and answer type.
02Reranker qualityGiven a fixed candidate set, measure whether relevant passages move upward using paired ranking metrics and score calibration where needed.
03Assembly retentionMeasure how often judged evidence survives token, source, duplicate, and position rules, with explicit omission reasons.
04Generation useTest supported answers, citations, abstention, contradiction handling, and sensitivity to removing or shuffling selected evidence.

End-to-end answer quality remains important, but it cannot locate the defect alone. Hold earlier stages fixed when evaluating a reranker; hold the reranked list fixed when evaluating packing. Otherwise a simultaneous index refresh, model update, and prompt change can create an apparent reranker win that no replay can attribute.

Budget latency, omissions, and rollback

Cross-encoder cost grows with candidate count and pair length. Observe candidate generation, queueing, reranking, tokenization, packing, model prefill, and generation separately, including timeout and cancellation rates. Tail latency matters because a degraded path may silently bypass reranking under load and create a different product behavior.

Log content-safe identifiers and policy reasons rather than unrestricted source text. Keep prior generator, reranker, tokenizer, packing policy, and prompt framing revisions available as one rollback unit. If a new tokenizer changes counts, rerun assembly and downstream evaluation; the old approval does not bind the new context bytes.

Operate at three altitudes

Production lens

— Track candidate recall, reranker lift, assembly retention, answer grounding, and tail latency as separate time series tied to exact revisions.
— Record deterministic omission reasons and treat tokenizer, context budget, source cap, and order as versioned serving policies.
— Define timeouts and degraded modes explicitly; a bypassed reranker or truncated context is a different evaluated path, not an invisible optimization.

Staff lens

— Assign ownership across retrieval, model inference, context policy, source governance, and evaluation so an end-to-end regression has a responsible boundary.
— Require launch and rollback records to cover the entire funnel, including score bindings and exact context bytes, rather than only a reranker checkpoint name.

Interview defense

A new cross-encoder improves NDCG on a fixed candidate set, but grounded answer accuracy does not move. How do you investigate?

I would keep stage identities explicit. First verify candidate recall on the production query slices; the reranker cannot recover absent evidence. Then bind and replay exact query-candidate pairs to confirm the ranking lift. Inspect whether token, duplicate, source-cap, or ordering rules discard the promoted evidence, and test whether the generator uses it through ablations and citation checks. I would also compare tail latency and degraded paths, because timeouts can erase offline gains.

Expect the interviewer to press on

— Why is score order insufficient without candidate identity?
— What changes when the serving tokenizer changes?
— How would you test a lost-in-the-middle effect?

Misconceptions to remove

“A stronger reranker fixes poor retrieval.”

It can improve ordering only within the candidates supplied. Missing relevant evidence is a candidate-generation or corpus problem.

“The top reranked chunks should simply be concatenated until the model limit.”

Packing also needs exact token accounting, duplication, source, provenance, framing, and position policies evaluated under the serving model.

“A long-context model will use all selected evidence equally.”

Usable attention can depend on position, distractors, prompt framing, and task. Test the actual context arrangement rather than inferring from the maximum window.

Check your model

1. Why must rerank evidence preserve candidate order if every candidate already has an ID?

A bare score vector is positional. Binding the ordered content IDs prevents a valid vector from being silently attached to a reordered or changed set.

2. What metric should be checked before blaming a reranker for a missing answer passage?

Candidate recall at the exact reranker input depth and production filters.

3. Does the artifact's source cap prove the selected context is diverse enough?

No. It enforces one declared count rule. Semantic diversity, source independence, and task adequacy require separate evidence.

Prove the mechanism

Add near-duplicate suppression using a separately versioned similarity policy. Preserve exact source/content identity and demonstrate a case where removing a duplicate makes room for necessary evidence without fabricating diversity.

Add a production constraint

Design a context-order experiment that randomizes the position of judged evidence while holding content fixed. Report task accuracy, citation use, latency, and uncertainty by position, model, and context length; define the policy change that the evidence would justify.

Artifact: Reranking and context assembly plan

courses/ai-engineering/reference-impl/reranking_context/reranking_context_audit.py

Download reference implementation

Primary references and next links

References

1. Passage Re-ranking with BERT
Nogueira and Cho. Primary example of query-passage cross-encoder reranking after candidate generation.
2. Dense Passage Retrieval for Open-Domain Question Answering
Karpukhin et al.. Primary dual-encoder retrieval work supporting the distinction between scalable candidate generation and later stages.
3. Lost in the Middle: How Language Models Use Long Contexts
Liu et al.. Primary study of position-sensitive use of relevant information in long contexts; behavior varies by task and model.
4. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al.. Primary RAG formulation connecting retrieved non-parametric evidence with generation.

Continue through the graph

Hybrid Retrieval and Score Fusion →
Produce a compatible broad candidate set before applying expensive relevance judgment.
Retrieval Security, Deletion, and Freshness →
Keep authorization, lifecycle, provenance, and untrusted-content boundaries intact through packing.

Glossary: candidate generation · cross-encoder · reranking · context assembly · token budget · source diversity · position effect