Failure labIntermediate90 min estimateOriginal publication

Decoding Is a Product Policy

The model emits logits. The product decides how to transform, constrain, sample, stop, retry, and expose them—and must version that decision like any other policy.

Authorship: InterviewsVector
Published / updated: 2026-08-25 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

A decoding policy begins with logits from a named model and tokenizer for one content-addressed request. It applies declared constraints, divides permitted logits by a positive temperature, computes stable softmax, sorts probabilities with a deterministic token-ID tie rule, retains the smallest prefix whose cumulative mass reaches top-p, renormalizes that support, and samples with a bound seed. It also binds maximum length, stop tokens, constraint versions, logits schema, and request scope. Reproducibility requires the exact logits record and policy contents—not just temperature=0.8. Changing any field creates a new policy identity. A reproducible selection is still not a quality, truth, safety, or user-preference guarantee.

Why this matters

Generation regressions are often blamed on a model release when a default temperature, tokenizer, stop token, schema constraint, seed path, or logits processor changed. If runtime selection is not content-addressed, teams cannot reproduce the support from which a token was drawn or tell model drift from policy drift.

You will be able to

Separate model logits from runtime selection policy.
Derive stable temperature scaling and nucleus support construction.
Define deterministic probability-tie and sampling conventions.
Bind model, tokenizer, request, logits, seed, stop, constraint, and policy identities.
Stress empty constraints, numeric extremes, stale scope, and content tampering.
Connect decoding changes to task, safety, structured-output, latency, and user evaluation.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Separate captured logits from constraints, transformation, truncation, random draw, stopping, and product fallback.
02
Derive
Derive stable temperature-scaled probabilities and the smallest deterministic top-p prefix.
03
Build
Build content-addressed policy, request, logits, support, draw, and decision records.
04
Stress
Vary temperature, top-p, ties, seed, token constraints, stops, scope, logits extremes, and generation bounds.
05
Operate
Evaluate policy cohorts by task, safety, refusal, validity, diversity, latency, cost, and user outcome.
06
Defend
Explain the exact reproduced selection and why it does not certify truth, quality, or safety.

Separate the model distribution from product policy

Logits are model outputs under a particular request and position. Temperature, top-p, allowed-token constraints, penalties, stop rules, maximum generation length, retries, and fallbacks transform how an application consumes them. A model comparison that changes these controls is also a policy comparison.

Layer	Owns	Version with
model	logits conditional on token context	model and architecture revision
tokenizer	token IDs and stop representation	vocabulary and normalization revision
decoder	temperature, truncation, draw, length	policy and RNG convention
constraint	allowed next tokens or schema state	grammar/schema revision
product	fallback, retry, refusal, presentation	release and experiment cohort

Temperature changes relative logit gaps

pᵢ(τ) = exp((zᵢ/τ) − m) / Σⱼ exp((zⱼ/τ) − m)

For positive τ, m is the maximum permitted scaled logit. Smaller temperature sharpens existing differences; larger temperature flattens them. Subtracting m prevents positive exponential overflow.

Temperature does not add knowledge or calibrate truth. It reshapes the distribution represented by one logits vector. Implementations disagree about whether temperature is ignored for greedy decoding, how zero is treated, and where penalties or constraints apply. Bind the order of operations, not just the number.

Top-p defines a support, not a creativity score

01Apply the declared constraintRemove forbidden token IDs before normalization under this policy version and fail closed if no candidate remains.
02Rank deterministicallySort by descending probability and then ascending token ID so equal logits have a stable order.
03Build the nucleusTake the smallest ranked prefix whose cumulative probability reaches or exceeds top-p.
04Renormalize and drawNormalize inside the retained support and use the bound seed, request, position, policy, and logits identities to derive the reproducible draw.
05Apply stop semanticsRecord whether the selected token is a declared stop; text-level stop strings require their own tokenizer-aware streaming contract.

Predict the nucleus before revealing it

Use fixed logits and choose a temperature and top-p policy, then predict which token IDs remain in the support. Check the prediction to reveal the stable-softmax and cumulative-mass calculation.

Predict the support admitted by temperature and top-p

Apply temperature to one fixed illustrative logit vector, then keep the smallest probability-ranked prefix whose cumulative mass reaches top-p. The probabilities and admitted support stay hidden until you check.

Temperature1.0Top-p threshold0.80

Fixed convention

Divide each logit by temperature.
Normalize with softmax and sort probability descending.
Include the token that crosses top-p, then stop.

Fixed logit fixture

Fixed illustrative token logits before temperature scaling
Candidate token	Raw logit	Probability / support
build	2.4	withheld until check
test	1.5	withheld until check
ship	0.6	withheld until check
retry	-0.2	withheld until check

Distribution shape and nucleus membership remain hidden while you reason from the logits and selected policy.

The vocabulary and logits are illustrative. This lab isolates support selection; it does not claim to reproduce any deployed model or sampling stack.

Which probability-ranked prefix remains eligible?

{build}{build, test}{build, test, ship}{build, test, ship, retry}

Make a prediction, then check it against the current evidence.

Build a reproducible policy decision

decoding_policy.py

1def apply_decoding_policy(
2    policy: DecodingPolicy,
3    request: DecodingRequest,
4    evidence: LogitsEvidence,
5) -> DecodingDecision:
6    """Apply the bound policy with stable ranking and a SHA-256-derived draw."""

Expected output

example=illustrative_only
policy_content_id=decoding-policy@sha256:bb86913520659d44f7c00af5d02e5be23a3c7aa28149181ca7cf95334f73711b
request_content_id=decoding-request@sha256:875f6bfe771cfd8dd1ec741ed99579c0ca062d1cf7d9a70fc905f67927909230
logits_content_id=decoding-logits@sha256:371366627090d54b083bc6e84adb712c5f147b78c00824cb6998789265edeea4
temperature=0.800
top_p=0.750
candidate_tokens=11,17
selected_token=11
selected_is_stop=false
status=PASS
claim=REPRODUCIBLE_POLICY_EXECUTION_NOT_QUALITY_GUARANTEE

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/decoding_policy

The excerpt is literal artifact source. The public function revalidates concrete immutable policy, request, and logits records; binds model, tokenizer, logits schema, request, constraint, stops, length, seed, and policy; rejects stale or tampered content IDs; caps prompt, vocabulary, token IDs, and generation length; and rejects booleans, non-finite, sub-resolution, or excessive logits and policy values.

Stress the entire selection path

Attack	Fail-closed behavior	Risk caught
constraint removes all IDs	refuse selection	invalid schema state
equal logits in new order	same token-ID tie order	iteration-order drift
NaN or infinite logit	reject evidence	undefined probability
stale tokenizer or request	reject scope	wrong token or context meaning
tampered logits under same ID	digest mismatch	unreproducible decision
position beyond max output	reject evidence	unbounded generation

Also test stop tokens split across text chunks, schema states with only one legal token, long repeated contexts, very sharp and flat distributions, content filters, tool-call envelopes, cancellation, retries, and fallback models. The small artifact handles token-ID constraints and stops only; it does not pretend to implement a full streaming grammar engine.

Evaluate decoding as a release policy

Outcome	Useful slices	Counter-metric
task success	task, locale, length, tool use	unsafe or invalid completion rate
diversity	prompt family and repeated runs	semantic consistency
structured validity	schema complexity and depth	fallback and repair rate
latency and cost	output length and stop reason	task completion
user preference	cohort and intent	factuality and safety review

Shadow a new policy on frozen logits to isolate selection changes before running live model traffic. Then compare controlled product cohorts with the same model and tokenizer. Log policy IDs, not raw sensitive prompts; retain secure replay references and decision aggregates. Roll back the policy independently of the model when its outcome gates fail.

Defend selection separately from generation quality

A passing artifact report supports one claim: the declared deterministic procedure reproduced one selected token from one content-addressed logits vector, request, and policy. It does not establish that the logits came from a correct model, that the token is factually true, that the sequence will satisfy a schema, or that users prefer the result.

Operate at three altitudes

Production lens

— Version model, tokenizer, logits processors, constraints, temperature, top-p, seed convention, stops, and generation limits together.
— Shadow policy changes on frozen logits before mixing them with model changes.
— Measure task success, safety, validity, diversity, latency, output length, stop reason, retries, and fallback by cohort.
— Keep privacy-safe replay identities that can reconstruct a decision under controlled access.

Staff lens

— Give model and decoding policy independent owners, release gates, and rollback paths.
— Require deterministic tie, processor-order, stop, and constraint semantics across runtimes.
— Separate reproducibility from quality and product quality from model likelihood.

Interview defense

How would you design and debug temperature and top-p decoding in production?

I would bind the exact model, tokenizer, request, logits schema, constraints, processor order, temperature, top-p, seed convention, stop IDs, and length bound. I would apply constraints, use maximum-subtracted temperature softmax, rank probability ties by token ID, retain the smallest cumulative top-p prefix, renormalize, and sample from a reproducible draw. I would replay frozen logits to isolate policy drift, then evaluate task, safety, validity, diversity, latency, and cost. Reproducibility does not prove quality or truth.

Expect the interviewer to press on

— What happens when a constraint removes every token?
— How do you make equal-probability ties deterministic?
— How would you distinguish a model regression from a policy regression?

Misconceptions to remove

“Temperature controls factuality.”

Temperature changes relative probabilities; factuality requires task evidence, grounding, and evaluation.

“A fixed seed is enough to reproduce generation.”

Model, tokenizer, request, logits, processor order, constraints, policy, position, runtime convention, and seed must all agree.

“Top-p keeps exactly p of the tokens.”

It keeps the smallest probability-ranked prefix whose cumulative mass reaches p; support size varies by distribution.

Check your model

1. Why does the policy sort equal probabilities by token ID before top-p truncation?

Without a declared tie rule, container or runtime iteration order can change the retained support and selected token.

2. Why should constraints be included in the content-addressed policy?

They change which tokens can receive probability and therefore change both the normalized distribution and decision.

3. What does stable softmax not protect against?

It does not validate stale logits, an incorrect model or tokenizer, unsafe policy choices, or downstream answer quality.

Prove the mechanism

Add an explicit top-k stage with a versioned processor order. Prove deterministic behavior for probability ties and show how top-k-before-top-p differs from top-p-before-top-k.

Add a production constraint

Design a streaming JSON-schema state machine with content-addressed grammar revisions, token-prefix legality, stop behavior, empty-support handling, and privacy-safe replay evidence.

Artifact: Decoding policy audit

courses/ai-engineering/reference-impl/decoding_policy/decoding_policy.py

Download reference implementation

Primary references and next links

References

1. The Curious Case of Neural Text Degeneration
Holtzman et al.. Primary paper motivating nucleus sampling and distinguishing likelihood from decoding behavior.
2. Locally Typical Sampling
Meister et al.. Primary work illustrating another distribution-shaped sampling policy and its evaluation.
3. Generation
Hugging Face Transformers documentation. Official operational reference for temperature, top-p, stopping criteria, and generation bounds.

Continue through the graph

The KV Cache Capacity Plan →
Connect output-length policy to reserved inference state.
Calibration, Thresholds, and Decision Cost →
Apply the same policy-versus-score distinction to predictive decisions.

Glossary: logits · temperature · top-p · nucleus sampling · constraint · stop token