Concept labIntermediate90 min estimateOriginal publication

When the Product Needs Ranking, Not Classification

Search, feeds, recommendations, and triage expose an ordered slate. Evaluate the positions users receive, not a bag of independent labels that forgot the candidate set and cutoff.

Authorship: InterviewsVector
Published / updated: 2026-08-20 / 2026-08-20
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Use a ranking objective and ranking evaluation when product value depends on relative order within a candidate set. Bind every observation to the query, ordered candidates, retrieval and index versions, model and features, cohort, evaluation window, judgment rubric, a bounded exact-integer grade-to-utility table, cutoff, unjudged-item rule, scale-stable aggregation with a non-negotiable safe weight-ratio floor, owners, and release thresholds. Compute DCG from contracted graded utility discounted by position, normalize against the ideal ordering under the same judgments, and gate both aggregate and per-query NDCG and label coverage. Classification accuracy, ROC AUC, or an aggregate mean can hide a useless top result, a failed query, missing judgments, numerical collapse, or a changed candidate generator.

Why this matters

A product rarely displays every positive item. It shows a few results in order, often after retrieval has already excluded most candidates. If evaluation ignores query boundaries, rank positions, candidate-set provenance, and label coverage, a model can improve an offline number while making the first screen worse. Ranking contracts make those product semantics executable.

You will be able to

Recognize when the unit of utility is an ordered list rather than an independent example.
Derive discounted cumulative gain and normalized DCG under an explicit grade-to-utility table.
Separate candidate generation, scoring, ordering, and presentation as distinct evaluation stages.
Bind query judgments, unjudged behavior, cutoff, aggregation, versions, owners, and aggregate plus per-query release floors.
Stress ranking evaluation for missing labels, duplicate evidence, position bias, and slice masking.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Represent each query as an ordered candidate list with scoped graded judgments and product cutoff.
02
Derive
Derive contracted gain, logarithmic rank discount, ideal DCG, NDCG, and judgment coverage.
03
Build
Implement a content-addressed evaluator with aggregate and per-query release floors.
04
Stress
Reorder results, remove judgments, reuse IDs, change candidate versions, and create a zero-utility query.
05
Operate
Monitor retrieval coverage, top-k utility, query slices, judgment maturity, and presentation effects together.
06
Defend
Explain why the metric matches the user surface and what logged behavior cannot identify by itself.

Choose the unit that matches the product surface

A ranking score cannot repair candidates that retrieval never supplied, and offline relevance does not establish downstream user impact.
Product question	Evaluation unit	Typical evidence
should this case trigger an action?	example and action	calibrated probability plus cost
which result should appear first?	query and ordered list	graded relevance at positions
did retrieval include useful items?	query and candidate set	candidate recall under a corpus snapshot
did the page help the user?	session and presentation	controlled outcomes with exposure

Independent classification discards competition. Predicting that ten documents are relevant does not say which four deserve the screen, while a top-k ranker must trade gain now against gain later. Preserve the candidate set and the exact order shown to make the metric reproducible.

Derive gain at the positions users receive

DCG@k = Σᵣ₌₁ᵏ gain(gradeᵣ) / log₂(r+1)

The contract maps each relevance grade to a non-negative utility and discounts later ranks. This artifact binds the complete table rather than assuming every team means the same transformation by DCG.

NDCG@k = DCG@k / IDCG@k

IDCG sorts the same available judgments by contracted utility. The evaluator rejects a query with no positive ideal gain instead of inventing a perfect or zero score.

Metric names are shorthand. Reproducibility lives in the convention fields.
Convention	Local choice	Consequence
grade utility	versioned bounded exact integers, illustrative (0,1,3,7)	adjacent utility levels cannot collapse when converted to binary64
discount	log₂(rank+1)	rank one is undiscounted; later utility decays
unjudged top-k item	zero gain and counts against coverage	missing labels cannot silently help
query aggregation	scale-stable query-weighted arithmetic mean	weights are normalized by their maximum and the contract floor cannot fall below 10⁻¹²

Build an evaluator that preserves every query

ranking_evaluation_contract.py

1        top_candidates = ranking.candidate_ids[: contract.cutoff]
2        top_grades = tuple(grade_by_candidate.get(item, 0) for item in top_candidates)
3        judged_in_top = sum(item in grade_by_candidate for item in top_candidates)
4        if not any(grade > 0 for grade in grade_by_candidate.values()):
5            raise ValueError("query has no positive relevance judgment")
6        ideal_grades = tuple(
7            sorted(grade_by_candidate.values(), reverse=True)[: contract.cutoff]
8        )
9        ideal_grades += (0,) * (contract.cutoff - len(ideal_grades))
10        ideal_dcg = _dcg(ideal_grades, contract.grade_gains)
11        if ideal_dcg <= 0.0:
12            raise ValueError("query has no positive ideal gain")
13        ndcg = _dcg(top_grades, contract.grade_gains) / ideal_dcg
14        coverage = judged_in_top / contract.cutoff

Expected output

example=illustrative_only
contract_version=ranking-evaluation-v1
metric=ndcg-at-k
cutoff=4
queries=2
mean_ndcg=0.921
judgment_coverage=1.000
decision=PASS

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/ranking_contract.

The excerpt is a literal contiguous source segment. The full artifact requires immutable tuples and concrete records; rejects booleans, invalid grades, utility gains above a safe exact-integer cap, non-finite or non-positive weights, weight ratios below a versioned bound whose value itself cannot fall below the global 10⁻¹² safety floor, ragged candidates, oversized inputs, and duplicate ranking, query, candidate, or judgment identities; binds evaluation, cohort, model, candidate-set, index, feature, labels, judgment rubric, utility table, numeric convention, metric conventions, cutoff, owners, and thresholds; and content-addresses every query ranking. It divides weights by their maximum before aggregation so equal subnormal weights cannot underflow the mean, while the ratio floor prevents a mixed-scale contribution from disappearing and bounded gains preserve adjacent rubric utilities with arithmetic headroom. Global judgment-ID reuse is rejected, and a passing aggregate cannot override a query below its NDCG or coverage floor.

Break aggregate comfort

01Create one zero-utility queryPair NDCG zero with a perfect query so the mean clears its threshold; require the per-query floor to hold the release.
02Move judgments outside top-kPreserve the candidate set but worsen positions. Classification counts stay unchanged while ranking utility falls.
03Remove top-k judgmentsConfirm unjudged candidates contribute zero gain and reduce coverage instead of disappearing from the denominator.
04Reuse a judgment identityCopy one judgment into another query and reject it globally, even though each query appears locally unique.
05Change retrieval lineageSwap candidate-set or index version while keeping scores and prove the comparison is invalid.
06Collapse adjacent utilitiesTry grade gains above binary64's exact-integer range and reject the contract before two distinct rubric levels can produce the same DCG.
07Lower the weight-ratio floorPair a unit-weight perfect query with a 5×10⁻³²⁴ imperfect query and reject a contract that tries to lower its own ratio floor below 10⁻¹².

Operate the whole retrieval-to-presentation path

Stage	Offline evidence	Production evidence
candidate generation	recall against corpus and judgment pool	empty-result and source coverage
ranking	NDCG, coverage, per-query and slice floors	score and position distributions
presentation	layout-aware review	exposure, latency, abandonment
outcome	mature utility labels	controlled experiment and guardrails
feedback	propensity or exploration contract	exposure-conditioned label coverage

Freeze query sampling, corpus, retrieval settings, deduplication, index, feature computation, ranker, judgment rubric, utility table, and cutoff for a comparison. Track head and tail queries, zero-result rates, latency, diversity, safety, and label coverage. Online experiments remain necessary when the product claim is user benefit rather than agreement with an offline judgment set.

Operate at three altitudes

Production lens

— Log query, candidate-set, index, feature, ranker, judgment-rubric, grade-utility, cutoff, and presentation identities.
— Report aggregate, per-query, and slice ranking metrics beside judgment coverage and query support.
— Monitor empty results, candidate recall proxies, latency, truncation, score distribution, diversity, and safety constraints.
— Treat interaction labels as exposure-conditioned evidence and preserve the policy that generated them.

Staff lens

— Assign separate ownership for corpus, retrieval, ranking, judgment quality, metric policy, experimentation, and presentation.
— Require the product team to state whose utility the grade table encodes and how disagreements or appeals are handled.
— Invest in query sampling and labeling coverage instead of optimizing a dense benchmark that omits difficult traffic.
— Review aggregate wins for per-query, cohort, language, safety, and latency regressions before release.

Interview defense

When would you use NDCG instead of classification accuracy, and what would you bind in the evaluation?

I would use a ranking metric when the product exposes an ordered top-k list and graded value depends on position. I would bind the query sample, ordered candidate set, retrieval and index versions, ranker and features, cohort and window, judgment rubric, grade-to-utility table, discount, cutoff, unjudged rule, query weights, owners, and thresholds. I would report candidate coverage separately, gate aggregate and per-query NDCG and label coverage, reject duplicate or stale evidence, and validate user outcomes online because offline judgments do not capture exposure and presentation effects.

Expect the interviewer to press on

— Why can accuracy remain constant while NDCG changes?
— How do unjudged documents affect your metric?
— Why is candidate generation evaluated separately?

Misconceptions to remove

“Ranking is classification followed by sorting probabilities.”

That can be a baseline, but ranking utility is query-relative and position-dependent; training and evaluation should preserve candidate competition and cutoff.

“Normalized DCG makes every query equally trustworthy.”

Normalization rescales gain, not judgment coverage, candidate difficulty, traffic value, or label bias.

“A higher mean NDCG means every query improved.”

Large gains can mask failed queries or slices. Preserve per-query metrics and enforce explicit floors or allowed-failure policy.

Check your model

1. Why does moving a relevant item from rank one to rank four matter?

The discount assigns less utility to later positions because users are less likely to receive or inspect them; the relevance labels alone are unchanged.

2. Why gate judgment coverage separately from NDCG?

Unjudged candidates create uncertainty. A plausible NDCG computed over sparse evidence cannot certify a list without a declared missing-label policy and support floor.

3. What does a candidate-set version protect?

It prevents a ranker comparison from silently changing the universe it was allowed to order, which could otherwise attribute retrieval movement to scoring.

Prove the mechanism

Create a small graded ranking set with at least four queries and a fixed cutoff. Bind query, candidate, index, feature, model, label, rubric, bounded grade-utility table, scale-stable query weights with a global ratio safety floor, cohort, window, owners, conventions, and release floors. Implement NDCG and judgment coverage, preserve content identities, and add adversarial tests for position changes, unjudged top-k items, one zero-score query hidden by the mean, equal and mixed-scale subnormal weights, an illegally lowered ratio floor, unsafe adjacent utility gains, duplicate IDs across queries, stale retrieval versions, invalid grades, ragged lists, booleans, and oversized evidence.

Add a production constraint

Add an exposure-aware counterfactual evaluation using logged propensities from a bounded randomized policy. State positivity assumptions, clip or reject unstable weights under a versioned rule, compare with a live experiment, and keep the ordinary relevance contract as a separate diagnostic.

Artifact: Ranking evaluation contract

courses/ai-engineering/reference-impl/ranking_contract/ranking_evaluation_contract.py

Download reference implementation

Primary references and next links

References

1. Cumulated Gain-based Evaluation of IR Techniques
Järvelin and Kekäläinen. Primary paper grounding graded gain, rank discounting, and normalized cumulated-gain evaluation.
2. The Relationship Between Precision-Recall and ROC Curves
Davis and Goadrich. Primary paper supporting metric choice for skewed decision problems and warning that ROC and precision-recall areas are not interchangeable objectives.
3. Optimizing Search Engines Using Clickthrough Data
Joachims. Primary paper supporting query-relative preference learning from logged ranking and click context.

Continue through the graph

Similarity Is a Retrieval Policy →
Connect vector scoring choices to the order ultimately evaluated.
Calibration, Thresholds, and Decision Cost →
Contrast ordered utility with probability-triggered actions.
Drift, Feedback Loops, and Delayed Labels →
Handle the exposure bias and delayed outcomes created by a live ranking.

Glossary: ranking · candidate set · graded relevance · DCG · NDCG · cutoff · judgment coverage · position bias