Failure labAdvanced120 min estimateOriginal publication

Precision and Quantization as Error Budgets

Choose a precision profile only after connecting tensor distributions, kernel coverage, calibration, boundary-case quality, memory traffic, and rollback to one versioned release contract.

Authorship: InterviewsVector
Published / updated: 2026-09-26 / 2026-09-26
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim device benchmarks, production capacity, statistical validation, hardware qualification, or universal model quality.

The decision in one pass

Precision is a tensor- and operation-specific error budget, not a model-wide adjective. Begin with the product tasks, boundary slices, calibration population, target devices, kernels, latency, throughput, memory, and cost constraints. For each weight, activation, KV, accumulator, reduction, normalization, and output path, reason about range, resolution, outliers, scaling granularity, clipping, rounding, overflow, underflow, and error accumulation. Choose a profile—mixed floating point, weight-only, weight-and-activation, static, dynamic, per-tensor, per-channel, or grouped—because supported kernels convert smaller representations into measured end-to-end value. Compare the exact candidate with a frozen higher-precision baseline on overall and harm-bearing slices, numerical probes, saturation, size, traffic, latency, throughput, energy, and telemetry. Release only the bounded device, shape, model, and task profile that meets predeclared budgets, then canary it with rapid fallback; never infer universal quality or speed from bit width alone.

Why this matters

Quantization can make a model fit, reduce memory traffic, and unlock efficient kernels, but it changes numerical behavior non-uniformly. Rare activation outliers can dominate a scale. A model can preserve average benchmark accuracy while failing a safety boundary. A nominal INT4 checkpoint can dequantize into a slower path on one backend, while another device accelerates it. Calibration data can leak, drift, or teach teams to optimize for the gate. Without a multi-dimensional release contract, a size win is easily mistaken for a product win.

You will be able to

Reason about range, resolution, scaling, clipping, rounding, accumulation, and outliers across integer and floating-point formats.
Choose weight-only, activation, KV, static, dynamic, per-tensor, per-channel, or group-wise schemes from tensor and kernel behavior.
Build representative calibration and evaluation evidence without using the release set as an optimization target.
Gate a versioned quantization profile on boundary quality, numerical error, saturation, size, throughput, and telemetry together.
Operate canary, fallback, drift detection, and requalification across model, data, kernel, compiler, and device changes.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map task and safety decisions to tensor roles, distributions, formats, scales, kernels, accumulators, devices, shapes, and fallback paths.
02
Derive
Derive representable range, quantization step, clipping and saturation behavior, error metrics, quality budgets, sample floors, and minimum system benefit.
03
Build
Build a deterministic integer gate over invented aggregate calibration, quality, size, and throughput evidence; do not quantize or benchmark a model.
04
Stress
Inject outliers, stale calibration, slice regressions, saturation, unsupported operators, conversion overhead, kernel drift, and misleading aggregate wins.
05
Operate
Monitor profile and backend versions, saturation, fallback, quality slices, latency, throughput, memory, cost, energy, telemetry health, and distribution drift.
06
Defend
Defend why the chosen precision is safe for this bounded profile, what evidence would invalidate it, and why lower bit width alone proves neither speed nor quality.

Allocate range and resolution where the computation needs them

Floating-point formats trade exponent range against significand resolution; integer quantization maps a real interval into discrete codes through a scale and often a zero point. The relevant error is not just reconstruction error for one tensor. Values flow through dot products, reductions, normalization, nonlinearities, residual additions, and repeated decoding steps. Accumulators may need more range than operands, and numerically sensitive operators may remain at higher precision even when most weights or activations are compressed.

q = clamp(round(x / scale) + zero_point); x_hat = scale · (q - zero_point)

Scale selection allocates finite codes across observed values. Clipping reduces range to improve resolution for common values but saturates tails; outliers can instead make most codes too coarse. Exact behavior also depends on rounding, signedness, granularity, and kernel implementation.

Choice	Potential benefit	Primary evidence needed
mixed FP16/BF16	mature tensor-core paths and smaller traffic	overflow, underflow, reductions, loss or output stability
weight-only INT8/INT4	weight capacity and bandwidth reduction	dequantization kernel, group size, outliers, task slices
weight-and-activation	larger traffic and compute opportunity	representative activation calibration and operator coverage
KV quantization	more context or concurrency	long-context and iterative error, cache kernel, attention quality
per-channel or grouped scales	better fit to heterogeneous distributions	metadata cost, kernel support, and grouping sensitivity

Calibrate the deployed computation, not an abstract checkpoint

01Define the profileBind model revision, tensor formats, scale granularity, group size, clipping method, calibration digest, operator exceptions, backend, compiler, kernels, device, and shapes.
02Sample calibration trafficCover languages, modalities, prompt lengths, domains, tool paths, policy boundaries, and rare but important activation patterns without leaking or retaining unnecessary content.
03Inspect tensor and layer evidenceMeasure tails, saturation, scale stability, per-layer error, sensitive operators, accumulated drift, and fallback or conversion paths rather than relying on one global norm.
04Keep release evidence independentDo not repeatedly tune on the final golden set. Preserve unseen tasks and boundary slices so the gate can disagree with the chosen profile.

Post-training quantization adapts scales or weights without retraining the full model; quantization-aware training exposes the model to simulated quantization during training. Neither label guarantees a result. The correct choice depends on model access, target profile, expected benefit, quality budget, training cost, reproducibility, and supported deployment kernels.

Gate one profile with an explicit error and benefit budget

quantization_release_gate.py

1def audit_quantization(contract: QuantizationContract, evidence: QuantizationEvidence) -> QuantizationReport:
2    contract = validate_record(contract, QuantizationContract)
3    evidence = validate_record(evidence, QuantizationEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another quantization contract")
6    # Evidence gaps hold. Quality, numerical, size, or throughput budget failures reject.

Expected output

example=illustrative_only
decision=ACCEPT_BOUNDED_PROFILE
accuracy_drop_bps=50;boundary_drop_bps=150
saturation_ppm=5000;compression_permille=3555;throughput_permille=1500
claim=LOCAL_QUANTIZATION_GATE_NOT_HARDWARE_OR_QUALITY_BENCHMARK

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/quantization_budget

The invented fixture compares a BF16 baseline with one INT4 weight-only profile on an invented backend. It reports a 50-basis-point aggregate drop, a larger 150-basis-point boundary-slice drop, 5,000 saturated values per million, 3.555× declared compression, and 1.5× declared throughput. The gate also checks sample floors, p99 absolute error, complete telemetry, exact counts, allowed formats, digest and scope binding, frozen records, type confusion, constructor bypass, and stale evidence.

The script does not quantize a model, inspect tensors, execute a kernel, validate labels, estimate statistical confidence, or reproduce any paper or device. Every value is invented. Acceptance applies only to the declared local profile and budgets; real release evidence needs uncertainty, task-specific harm analysis, representative devices, and canary observation.

Gate a bounded quantization profile

Compare invented overall and boundary quality, saturation, size, and throughput evidence against a declared profile. Predict accept, hold, or reject before checking.

Spend precision only where the error budget permits

Treat numeric format as a release contract across calibration coverage, quality slices, runtime support, memory savings, and latency. Predict the defensible disposition before revealing it.

Quantization record

Calibration misses the serving distributionThe scale set was collected from short English prompts, but the release serves mixed long contexts.One safety-sensitive head exceeds its error budgetAggregate quality holds, but a narrow output path changes high-impact decisions.The comparison uses a stale baselineThe candidate identity is correct, but quality deltas reference an earlier full-precision revision.Every declared error budget holdsRepresentative calibration, critical slices, resource savings, and runtime support all match the contract.

Formats, models, scores, and fleet results are invented for practice. Passing this lab is not a model-quality or hardware certification.

Error-budget evidence
Signal	Operating constraint	Observed evidence
Calibration set	Match declared serving slices	English only; 128–512 tokens
Serving mix	Bound to the release contract	6 languages; up to 16k tokens
Activation clipping	Below 0.5% per declared slice	4.7% on long-context slice
Memory reduction	At least 35%	47%

Illustrative measurements only. The recommended action stays hidden until you check a prediction.

What should happen to this quantized candidate?

Admit to a bounded serving reviewRecalibrate on representative activationsKeep the sensitive path at higher precisionReject the evidence as out of contract

Choose an action before checking the operating contract.

Require both numerical safety and system value

Evidence plane	Release question	Failure hidden by averages
task quality	does the candidate preserve product decisions?	rare language, long context, structured output, or safety boundary loss
numerical	where do clipping, saturation, and error accumulate?	one sensitive layer or operator behind a small global norm
kernel	does the exact backend execute the intended format?	dequantization, fallback, conversion, or unsupported shape
system	did memory, traffic, latency, throughput, cost, or energy improve?	faster microkernel but unchanged queueing or end-to-end latency
operations	can the profile be attributed, monitored, and rolled back?	mixed revisions or silent higher-precision fallback

Use paired outputs and task decisions to localize regressions, then segment by tensor profile, sequence shape, device, language, domain, and risk boundary. A candidate that improves throughput but violates a hard safety slice is rejected; a candidate with insufficient sample or broken telemetry is held; a candidate that passes earns only a bounded canary with fallback.

Operate precision as a versioned serving profile

Store the quantization manifest beside the model: source revision, method, formats by tensor or operator, scales or scale-generation procedure, group size, calibration provenance and digest, exceptions, backend, compiler, kernel, device, evaluation, and rollback target. Emit the active profile in decision traces. Detect silent fallback to another precision and keep the higher-precision path deployable until canary and delayed outcomes are understood.

Monitor saturation and activation drift on privacy-bounded statistics tied to the deployed profile and population.
Track quality and safety slices, invalid outputs, fallback, kernel coverage, memory, traffic, latency, throughput, cost, energy, and telemetry completeness.
Requalify after model adaptation, tokenizer, prompt distribution, context length, compiler, kernel, driver, hardware, or scheduler changes.
Canary profile changes by stable assignment and preserve immediate rollback; do not mix baseline and candidate evidence under one version label.

Operate at three altitudes

Production lens

— Version and trace the complete precision profile, calibration digest, backend, compiler, kernel, device, model, and shape; alert on silent operator fallback or mixed revisions.
— Monitor product and safety slices beside saturation, numerical probes, invalid values, size, traffic, latency, throughput, cost, energy, and telemetry health.
— Keep a tested higher-precision rollback and requalify the candidate when tensor distributions, model behavior, serving software, or target hardware changes.

Staff lens

— Make model, evaluation, safety, compiler, kernel, serving, hardware, and data-governance owners agree on one bounded profile and release contract rather than optimizing bit width in isolation.
— Require a minimum system benefit and a maximum quality or harm budget so quantization complexity is justified by product-level evidence, not checkpoint size alone.

Interview defense

An INT4 model is four times smaller and passes the average benchmark, but one safety slice regresses and production latency does not improve. Do you ship it?

No. The safety slice is a hard boundary, and the missing end-to-end benefit means the profile has not justified its added complexity. I would verify the exact model, calibration, group size, backend, kernels, device, and shapes; inspect saturation, outliers, sensitive layers, operator fallbacks, conversion, and whether the service is limited by another resource. I would keep the release set independent, test task and harm-bearing slices with uncertainty, and compare memory traffic, latency, throughput, cost, and energy on the deployed path. A revised mixed profile might keep sensitive operators at higher precision, but it still needs a versioned canary and fast higher-precision fallback.

Expect the interviewer to press on

— Why can per-channel scales help and what do they cost?
— How can a smaller checkpoint fail to reduce latency?
— Which changes invalidate calibration evidence?

Misconceptions to remove

“Quantization error is small if the average tensor reconstruction error is small.”

Small average error can hide sensitive layers, tails, saturation, accumulated error, structured-output failures, or task and safety regressions. Release evidence must follow product decisions.

“INT4 is always twice as fast as INT8 and four times as fast as FP16.”

Bit width changes storage opportunity, not guaranteed wall-clock speed. Kernel support, packing, dequantization, operator coverage, shape, traffic, and the active bottleneck determine measured benefit.

“One calibration set qualifies a profile for every deployment.”

Activation distributions, model revisions, tasks, prompts, context lengths, populations, kernels, devices, and compiler behavior can change the valid range of a profile.

Check your model

1. Why do outliers create a scaling trade-off?

Covering them with one scale expands range but coarsens resolution for common values; clipping them improves common-value resolution but saturates the tail.

2. What makes a quantization release profile bounded?

It binds exact model, tensor and operator formats, scales or calibration, backend, kernels, device, shapes, tasks, slices, benefits, budgets, monitoring, and rollback.

3. Why require a minimum throughput or memory benefit?

Quantization adds calibration, evaluation, operational, and debugging complexity; without measured system value, it creates risk without solving the product constraint.

Prove the mechanism

Extend the artifact with two named risk slices and an operator-fallback count. A fallback or either hard slice must reject even when aggregate quality and compression pass; insufficient slice samples must hold.

Add a production constraint

Design a mixed precision profile for weights, activations, KV, accumulators, normalization, and output heads across two device types. Include calibration governance, kernel matrix, quality uncertainty, boundary slices, numerical probes, traffic, latency, energy, canary, drift, and rollback.

Artifact: Quantization error-budget release gate

courses/ai-engineering/reference-impl/quantization_budget/quantization_release_gate.py

Download reference implementation

Primary references and next links

References

1. Working with Quantized Types
NVIDIA TensorRT. Official deployment guidance on explicit quantization, scales, calibration, precision, and quantized operator behavior.
2. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Xiao et al.. Primary research on shifting activation outlier difficulty into weights for post-training W8A8 quantization.
3. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar et al.. Primary research on one-shot post-training weight quantization for large language models.

Continue through the graph

Numerical Stability Is Part of the Algorithm →
Connect representational and accumulation error to bounded product behavior.
The Accelerator Memory Hierarchy →
Verify that a smaller representation reduces traffic at the active device boundary.

Glossary: quantization · scale · zero point · clipping · saturation · per-channel quantization · group-wise quantization · post-training quantization · quantization-aware training · mixed precision