AI/ML Interview Knowledge Hub

Direct answer

Offline results are estimates under a historical dataset and evaluation protocol; production introduces a different population, delayed labels, feedback loops, system failures, and business costs. I would first verify leakage and split quality, then compare feature and prediction distributions by cohort, reproduce the online preprocessing path, and measure the decision outcome—not just the model score.

Mechanism

A model can be statistically healthy on an unrepresentative test set while the surrounding product is unhealthy. The useful mental model is a chain: raw event → feature → prediction → decision → user response → delayed label. Any break or distribution change along that chain can create an offline/online gap.

Direct answer

High bias means the hypothesis or features are too restrictive to capture the signal, so both training and validation error stay high. High variance means the model fits unstable details of the training sample, so training error is low but validation error is much worse. I diagnose the pattern with learning curves, then change model capacity, regularization, features, or data accordingly.

Mechanism

Bias and variance are not labels for model families; they describe error behavior under repeated samples. More data often reduces variance but does not repair missing signal or a fundamentally biased objective.

Direct answer

Leakage occurs when training or evaluation uses information unavailable at the real prediction time, including future events, target-derived features, duplicated entities, or preprocessing fitted across splits. I define the prediction timestamp and entity boundary first, build splits that respect them, fit transformations only on training data, and audit suspiciously strong features and performance jumps.

Mechanism

The governing question is not ‘is this column allowed?’ but ‘could the serving system know this value at the decision timestamp?’ Point-in-time joins and group-aware splits turn that question into enforceable data contracts.

Direct answer

Regularization biases learning toward simpler or more stable solutions, such as smaller weights, sparse features, or robustness across perturbed examples. It helps when variance is the problem, but too much increases bias and can erase weak real signal. I tune it on validation data and inspect per-slice behavior rather than assuming a larger penalty is safer.

Mechanism

L1 can encourage sparsity, L2 penalizes large weights smoothly, and deep-learning techniques such as dropout, augmentation, and early stopping regularize through different mechanisms. The right choice depends on model and data behavior.

Direct answer

Linear regression models a conditional numeric expectation and is commonly fit with squared-error objectives. Logistic regression models log-odds as a linear function, maps them through a sigmoid, and is trained with a probabilistic classification loss. Both are linear in their features, so interactions and nonlinear structure must be represented explicitly.

Mechanism

The useful comparison includes assumptions, loss geometry, calibration, and decision thresholds—not merely that one predicts numbers and the other predicts classes.

Direct answer

A single tree is easy to inspect but unstable and usually lower quality. A random forest reduces variance by averaging decorrelated trees and is a strong robust baseline. Gradient boosting fits errors sequentially and often wins on structured tabular data, but needs more careful tuning and can be sensitive to leakage, noisy labels, and distribution shift.

Mechanism

I choose after considering dataset size, missing values, monotonic constraints, latency, calibration, interpretability, and operational tooling—not benchmark rank alone.

Direct answer

SVMs fit medium-sized, high-dimensional problems with a meaningful margin and carefully scaled features. KNN is useful when local similarity is meaningful and inference volume is modest, but suffers in high dimensions. Naive Bayes is a fast, data-efficient baseline for sparse counts or text despite its conditional-independence assumption.

Mechanism

The interview signal is whether you connect each algorithm's inductive bias and computation to the data rather than dismissing older methods categorically.

Direct answer

PCA finds orthogonal directions of maximum linear variance; it compresses or denoises features but does not discover semantic groups. Clustering proposes groups from a distance and shape assumption, such as spherical clusters for k-means. Both can produce convincing structure from scaling artifacts, mixed units, outliers, or a distance metric that does not match the product.

Mechanism

Fit preprocessing on the training population, inspect stability across resamples, and validate clusters against downstream usefulness rather than a plot alone.

Direct answer

Precision asks how many predicted positives were correct; recall asks how many actual positives were found. The right balance comes from the relative cost of false positives and false negatives at a chosen threshold. F1 is a compact harmonic mean when both matter, but it hides calibration, prevalence, true negatives, and asymmetric business cost.

Mechanism

Report the confusion matrix and metric by meaningful slices. A single aggregate can improve while the most important cohort degrades.

Direct answer

ROC-AUC measures ranking across true-positive and false-positive rates and can look strong when negatives dominate. PR-AUC focuses on precision and recall for the positive class, so it is usually more revealing for rare-event retrieval or detection. Neither selects an operating threshold or captures the real cost of errors.

Mechanism

Always compare against the class-prevalence baseline and inspect the part of the curve where the system can actually operate.

Direct answer

A calibrated score of 0.8 should correspond to roughly an 80% event rate for comparable cases. Calibration matters when probabilities drive pricing, triage, risk, or expected-value decisions; ranking quality alone is insufficient. I assess reliability by slice on held-out data, recalibrate without reusing training predictions, and select thresholds from cost and capacity constraints.

Mechanism

Calibration can drift even when ranking remains stable, especially after prevalence or policy changes. Monitor both discrimination and reliability.

Direct answer

For ranking, choose a metric that matches position value and relevance: MRR emphasizes the first relevant result, Recall@k candidate coverage, and nDCG graded relevance near the top. For regression, MAE is robust and directly interpretable, while RMSE penalizes large errors more. I also report slices and a product outcome because no offline metric fully represents user value.

Mechanism

Metric choice encodes a utility function. State which errors matter, at what positions or ranges, and which constraints must remain guardrails.

Direct answer

Offline evaluation gives fast reproducible evidence on frozen datasets; shadow traffic tests the real request path without affecting decisions; canaries limit exposure; and controlled experiments measure user and business outcomes. Each stage should have explicit quality, safety, latency, and cost gates, plus a rollback. Passing one stage does not substitute for the next.

Mechanism

The hierarchy separates scientific comparison from operational readiness and causal product impact. Preserve the exact model, prompt, data, and policy versions across the release record.

Direct answer

The forward pass computes activations and a scalar loss. Backpropagation applies the chain rule from that loss backward through the computation graph to compute each parameter's contribution to the error. An optimizer then uses those gradients to update parameters; backprop computes gradients, while the optimizer decides the step.

Mechanism

Reverse-mode automatic differentiation is efficient because many parameters influence one scalar loss. Saved activations trade memory for the ability to compute local derivatives during the backward pass.

Direct answer

Deep compositions repeatedly multiply local derivatives; values consistently below one shrink the signal, while large values amplify it. I inspect gradient norms and activation distributions by layer, reproduce the first unstable step, then test learning rate, initialization, normalization, residual paths, precision, and data outliers. Clipping can contain explosions but should not hide the root cause.

Mechanism

The diagnosis is temporal and layered: identify which batch and which layer becomes non-finite first. Later NaNs are symptoms, not independent failures.

Direct answer

Normalization stabilizes the scale and distribution seen by later computations, making optimization easier; batch normalization uses batch statistics, while layer normalization works within each example and fits sequence models well. Dropout randomly removes activations during training to discourage brittle co-adaptation. Their value and placement depend on architecture, batch regime, and train/inference behavior.

Mechanism

Neither is a universal quality switch. Batch norm can be problematic with tiny or shifting batches, and dropout that is too strong can reduce capacity or interact poorly with fine-tuning.

Direct answer

The learning rate usually dominates optimization stability: too high diverges or oscillates, too low wastes compute or settles poorly. Momentum smooths and accelerates consistent directions; adaptive methods scale steps per parameter and often reach a useful model faster. I select them with loss curves, gradient norms, validation behavior, batch size, and a schedule—not by optimizer name alone.

Mechanism

Warmup can protect early training before activation and optimizer statistics stabilize, while decay shifts from rapid progress toward refinement. Reproduce comparisons at equal compute and tuned learning rates.

Direct answer

Self-attention lets every token build a context-dependent representation by directly weighting other positions, avoiding the long recurrent path that makes distant dependencies and parallel training difficult. Standard attention exposes broad parallelism during training, but its score matrix grows quadratically with sequence length. Autoregressive generation remains sequential token by token.

Mechanism

The key improvement is path length and parallel computation, not that attention is free or that order disappears. Positional information and masking restore sequence constraints.

Direct answer

A token's query represents what contextual information it seeks; keys describe what positions offer; their scaled dot products produce weights over values, which carry the information to aggregate. Multiple heads learn separate projection subspaces, allowing different relationships to be represented in parallel before their outputs are combined.

Mechanism

Heads are not guaranteed to map cleanly to human concepts, and more heads do not automatically improve quality. Head count changes dimensions, compute, KV-cache shape, and parallelism.

Direct answer

Attention alone is permutation-equivariant, so models need positional information to distinguish order. A causal mask prevents a decoder from reading future tokens during next-token training, while padding masks exclude non-content positions. Relative and rotary schemes encode distance differently and affect how a model behaves beyond its training context.

Mechanism

Context extension is not just raising a configuration value: position method, attention implementation, memory, data, and evaluation must all support the longer sequence.

Direct answer

Models consume tokens, not characters or words, so language, code, whitespace, and identifiers can expand into very different sequence lengths. Context limits constrain instructions, history, retrieved evidence, and output; longer inputs increase memory, prefill work, latency, and cost. I budget context by value and reserve explicit space for the response.

Mechanism

Truncation must preserve system constraints and the most useful evidence rather than simply dropping the oldest text. Token counts should be measured with the deployed tokenizer and model configuration.

Direct answer

Temperature rescales logits before sampling, changing how peaked the distribution is. Top-k restricts choices to a fixed number of tokens; top-p keeps the smallest set whose cumulative probability reaches a threshold. These controls change diversity and repeatability, not factual grounding, permissions, or correctness guarantees.

Mechanism

Use deterministic or constrained decoding where contracts matter, and evaluate settings on the actual task. Lower temperature cannot repair missing evidence or an unsafe tool boundary.

Direct answer

Prefill processes the prompt in parallel and builds key/value states for each layer; decode generates new tokens sequentially while reusing those cached states. Prefill is compute-heavy and grows with input length, while decode often becomes memory-bandwidth and KV-capacity bound. This distinction drives batching, scheduling, and separate time-to-first-token versus inter-token-latency SLOs.

Mechanism

The KV cache avoids recomputing earlier tokens but consumes accelerator memory proportional to active sequence length, layers, precision, and key/value head configuration.

Direct answer

An embedding maps an input into a dense vector so distances or directions encode relationships learned from the model's training objective. Similarity therefore means ‘similar according to this model and representation task,’ not universal semantic truth. I validate it on labeled queries, hard negatives, languages, domains, and the exact retrieval objective.

Mechanism

Two vectors can be close for topical, lexical, stylistic, or task-specific reasons. Product meaning comes from the embedding model, preprocessing, pooling, similarity function, and evaluation set together.

Direct answer

Cosine compares direction and ignores magnitude; dot product includes magnitude; Euclidean measures geometric distance. If vectors are unit-normalized, cosine ranking and dot-product ranking are equivalent, and squared Euclidean is closely related. The correct choice is the metric used or recommended by the embedding model and verified on retrieval data.

Mechanism

Changing normalization or metric after indexing changes neighborhood structure. Store those choices with the embedding version and test migrations end to end.

Direct answer

I start with query and corpus languages, modality, domain, maximum input length, latency, throughput, hosting, privacy, and dimension constraints. Then I benchmark candidate models on a labeled in-domain retrieval set with hard negatives and slice metrics, measure serving and index cost, and plan versioned dual-index migration before committing.

Mechanism

Public leaderboards are candidate generators, not product evidence. A higher-dimensional model may improve recall but increase storage, memory bandwidth, indexing time, and migration cost.

Direct answer

Dense retrieval optimizes learned similarity, so it can overvalue broad topical resemblance while missing exact identifiers, negation, freshness, permissions, or the user's actual intent. I inspect labeled failure cohorts, raw candidate ranks, chunk boundaries, metadata filters, model/version parity, and lexical baselines before changing the generator.

Mechanism

Hybrid retrieval and reranking solve different stages: lexical search restores exact signals; reranking improves ordering among candidates. Neither can recover documents that were not indexed or were filtered incorrectly.

Direct answer

RAG retrieves relevant external evidence at request time and gives that evidence to a generator, so knowledge can be updated and cited without retraining facts into model weights. A production system is two versioned pipelines: ingestion builds authorized searchable representations, and the query path retrieves, reranks, assembles context, generates, validates citations, and can abstain.

Mechanism

RAG reduces reliance on parametric memory but does not guarantee truth. Retrieval can be stale or wrong, and the generator can ignore or misread good evidence.

Direct answer

Use RAG to supply changing, private, attributable knowledge at inference time; use fine-tuning to change stable behavior, format, domain language, or task performance. RAG adds retrieval latency and index operations, while fine-tuning adds dataset, training, evaluation, and model-lifecycle cost. They can be combined when both knowledge access and behavior adaptation are justified.

Mechanism

The misleading shortcut is ‘fine-tune the model on our documents.’ Fine-tuning does not provide reliable, updateable factual storage or access-controlled citation by itself.

Direct answer

Chunks must be small enough to retrieve a focused answer but large enough to preserve meaning, provenance, and surrounding constraints. Fixed token windows are a baseline; structure-aware boundaries, parent-child retrieval, overlap, and metadata often work better. I tune chunking jointly with the embedding model, query set, reranker, context budget, and citation requirements.

Mechanism

Chunk size changes candidate count, index size, recall, reranker work, and context duplication. There is no universally best token count.

Direct answer

Lexical retrieval protects exact names, codes, and rare terms; dense retrieval finds semantic paraphrases. Their raw scores are not necessarily comparable, so combine ranked lists with a method such as reciprocal rank fusion, then apply a stronger reranker to a bounded candidate set. Evaluate recall before reranking and precision after it.

Mechanism

First-stage retrieval optimizes coverage cheaply; reranking spends more compute to improve order. Reranking cannot recover a relevant passage absent from the candidate set.

Direct answer

Evaluate the earliest possible stage: ingestion coverage and freshness, Recall@k for candidate retrieval, rank metrics after reranking, context precision and recall, then answer correctness, groundedness, citation validity, and abstention. Track latency, failures, tokens, and cost separately. A single end-to-end score cannot tell the team what to repair.

Mechanism

Freeze representative queries, expected evidence, acceptable answers, and hard negatives. Add production incidents and changed-corpus cases to the regression set with the full version bundle.

Direct answer

I compare the last good and current version bundle, then replay a frozen query set through each stage: corpus coverage, parsing, chunking, ACL metadata, embeddings, indexes, filters, candidate retrieval, fusion, reranking, context assembly, prompt, and generator. I identify the earliest changed output, mitigate with rollback or a safe bypass, and verify both quality and isolation before ramping again.

Mechanism

Fluent bad answers tempt teams to tune the prompt first. Stage-level replay keeps the investigation causal and prevents generation changes from masking a retrieval failure.

Direct answer

Full fine-tuning offers maximum parameter freedom but requires much more accelerator memory, optimizer state, storage, and release discipline. LoRA learns low-rank updates to selected projections, often reaching strong task quality with far fewer trainable parameters and portable adapters. I choose from measured quality, multi-task interference, serving topology, and operational complexity.

Mechanism

PEFT reduces training cost, not the need for data governance, evaluation, model licensing review, or serving validation. Adapter composition and base-model version compatibility become lifecycle concerns.

Direct answer

RLHF typically trains a reward model from preference data and then optimizes the policy against that learned reward with constraints such as a KL penalty. DPO turns preferred-versus-rejected pairs into a direct classification-style objective relative to a reference policy, avoiding a separate reward-model and online reinforcement-learning stage. DPO is operationally simpler; RLHF can support richer reward signals and online exploration but adds instability, infrastructure, and reward-hacking risk.

Mechanism

Both methods learn a policy from human preference evidence; neither proves factuality or safety. The release decision still needs preference-data governance, held-out behavioral and safety evaluations, slice analysis, and regression checks against the base model.

Direct answer

The dataset should represent the target task and failure costs, use consistent instructions and outputs, remove leakage and duplicates, preserve hard and refusal cases, document provenance and consent, and hold out entity- and time-safe evaluation sets. I prefer a smaller reviewed set over a large inconsistent scrape, then inspect slice learning and regressions against the base model.

Mechanism

Training data is a behavioral specification. Contradictory examples, hidden templates, contamination, and unbalanced easy cases can dominate the learned policy.

Direct answer

A narrow or repetitive dataset can push the model toward one behavior and degrade broader capabilities, while excessive steps or learning rate can overwrite useful representations. I compare task and general-capability suites, safety behavior, calibration, and slice quality throughout training; use lower-rank or smaller updates, replay data, regularization, and early stopping; and retain the base-model fallback.

Mechanism

A lower training loss is not sufficient evidence. The release unit includes the adapter or weights, base model, tokenizer, prompt, decoding policy, and evaluation results.

Direct answer

Do not use an agent when the steps, data dependencies, and failure handling can be encoded as a deterministic workflow. Model-directed control is justified when the path genuinely depends on ambiguous observations and bounded adaptation adds measurable value. The agent must still operate inside explicit permissions, budgets, termination rules, validation, and human-approval boundaries.

Mechanism

An agent is a control-loop choice, not a synonym for an LLM feature. Each extra autonomous step compounds latency, cost, nondeterminism, and attack surface.

Direct answer

The model proposes a typed action from a bounded catalog; deterministic code validates schema, identity, authorization, policy, budgets, and idempotency before execution. The tool returns a structured observation, which updates durable state for the next step. The loop ends on success, explicit failure, no progress, time, step, or cost limits.

Mechanism

The model is a planner or selector, never the enforcement point. Separate proposal, policy, execution, observation, and audit so failures are attributable.

Direct answer

Durable state records facts the workflow must recover—goal, steps, tool outcomes, approvals, versions, and side effects. Context is the bounded view selected for the next model call. Long-term memory should be explicit, sourced, scoped, expirable, and permission-aware; a transcript or vector store is not automatically trusted memory.

Mechanism

Separate authoritative business state from model-generated summaries. Store references and provenance so context can be reconstructed, corrected, and audited.

Direct answer

Use multiple agents only when separate roles need different permissions, contexts, models, lifecycles, or independent scaling and evaluation. Decomposing one prompt into personalities rarely justifies coordination cost. Start with a deterministic orchestrator and specialized workers, define message and state contracts, and measure whether decomposition improves quality enough to offset latency, cost, and failure modes.

Mechanism

Organizational metaphors are not architecture. The boundaries must correspond to real capability, security, or operating differences.

Direct answer

First stop or quarantine side-effecting execution, then inspect the trace: model proposal, tool schema, arguments, result, state update, retry policy, and termination decision. Common causes are ambiguous success responses, lost state, automatic retries at multiple layers, impossible goals, or missing no-progress detection. Add deduplication, bounded retries, explicit terminal outcomes, and regression replay.

Mechanism

The key evidence is whether the model ignored a successful observation or the system failed to persist and present it. Those require different fixes.

Direct answer

Define task-specific dimensions such as correctness, groundedness, instruction following, safety, style, and abstention; build a versioned dataset from real distributions, hard cases, and incidents; and combine deterministic checks, model-based graders calibrated to humans, and targeted human review. Gate releases on slice-level quality plus latency, cost, and safety, then monitor online outcomes and feed failures back into the set.

Mechanism

Evaluation is a living product interface. Every example needs provenance, expected evidence or rubric, and enough context to reproduce the exact model, prompt, tool, retrieval, and policy bundle.

Direct answer

A model judge scales rubric-based comparison when exact-match metrics cannot represent valid answers. It can favor verbosity, style, position, or models similar to itself and can be fooled by unsupported confidence. I use blinded pairwise or rubric scoring, randomize order, require evidence, calibrate against human labels by slice, and keep deterministic checks for facts and constraints.

Mechanism

Judge agreement is not ground truth. Track correlation and disagreement with expert reviewers, and version the judge prompt and model like any other evaluator.

Direct answer

A golden set makes high-value cases reproducible and enables fast regression checks; human evaluation handles ambiguity, evolving preferences, and cases where the rubric itself is uncertain. Sample humans strategically for disagreements, high-risk outputs, new cohorts, and judge calibration rather than reviewing everything uniformly.

Mechanism

Measure reviewer agreement and give adjudication rules. If experts cannot apply a rubric consistently, an automated judge will not rescue it.

Direct answer

Measure task completion and downstream outcomes alongside explicit feedback, corrections, abandonment, retries, escalation, safety events, latency, and cost. Interpret implicit signals carefully because users adapt to the system and negative outcomes may be silent. Use randomized experiments or causal designs where safe, plus review channels for high-impact decisions.

Mechanism

A thumbs-up rate is selection-biased and incomplete. Connect requests to outcomes and cohorts without collecting more sensitive content than the evaluation requires.

Direct answer

Stable application policy belongs in controlled system instructions; user intent belongs in the user message; retrieved or tool-produced content is untrusted data with explicit delimiters and provenance. Deterministic code must enforce permissions and output constraints because text placement cannot turn untrusted content into a safe authority boundary.

Mechanism

The prompt should make trust roles legible, but security comes from out-of-model policy, tool mediation, validation, and least privilege.

Direct answer

Schema-constrained output reduces parsing ambiguity and makes model responses easier to test and route, but schema validity does not guarantee semantic correctness, authorization, safe values, or an idempotent side effect. Validate ranges and invariants, look up authoritative entities, enforce policy, and treat the result as a proposal before execution.

Mechanism

Separate syntactic conformance from domain validation. A perfectly valid JSON object can still request the wrong account or an unsafe operation.

Direct answer

Context should contain the minimum authoritative evidence and state needed for the next decision, ordered and labeled so conflicts are visible. Tool descriptions are API contracts: capability, required fields, side effects, permission scope, failure modes, and when not to use the tool. More context or more tools can reduce quality by increasing ambiguity and attack surface.

Mechanism

Measure tool-selection accuracy and context ablations. If removing a context source improves quality, the problem may be selection or conflict, not model capacity.

Direct answer

Exact search scores every vector; ANN searches a smaller candidate region. HNSW navigates a layered proximity graph and offers strong recall with memory and update costs. IVF assigns vectors to coarse partitions and probes selected lists, often pairing with compression for scale. Search breadth parameters trade latency and compute for recall, so benchmark them on the real distribution.

Mechanism

Index choice also depends on filtering, write rate, delete behavior, persistence, sharding, and operational maturity—not only query latency.

Direct answer

Authorization and hard eligibility constraints must be enforced before or inside candidate selection; post-filtering a global top-k can leak information and destroy recall. Hybrid search runs lexical and dense retrieval over the authorized corpus, then fuses ranks and optionally reranks. The index must support the filter selectivity and update pattern without unpredictable tail latency.

Mechanism

Soft preferences may belong in ranking, but security, tenancy, region, and legal constraints are not scoring hints.

Direct answer

Build a new versioned index from canonical documents, dual-write live changes, backfill and reconcile counts, ACLs, deletes, and freshness, then shadow representative queries across old and new. Compare recall, ranking, latency, and isolation before canarying reads through an alias. Keep the old index until rollback and cache-expiry windows close.

Mechanism

Never mix incompatible dimensions or vector spaces in one logical index. Cache keys and trace metadata must include embedding and index versions.

Direct answer

Partition by tenant, region, semantic domain, or a balanced hash according to isolation and query locality; replicate for availability and read scale; and bound fan-out with routing metadata. Measure global Recall@k after merge, not shard-local recall alone, and protect p99 with admission, bounded search breadth, timeouts, and a tested degraded mode.

Mechanism

Sharding changes the statistical search problem as well as operations. Over-partitioning can require wide fan-out, while hot tenants or domains may need dedicated placement.

Direct answer

Record immutable references to code, data snapshot and split, feature logic, configuration, dependencies, container or environment, hardware, random seeds, tokenizer, initialization, and every produced artifact. Deterministic kernels help but exact floating-point identity is not always practical; the real contract is reproducible inputs, bounded variance, comparable metrics, and traceable lineage.

Mechanism

A model file without its training context is not reproducible. The run record should answer which data and code produced it and whether the result stays inside an expected distribution when rerun.

Direct answer

The release unit includes model or adapter, base model, tokenizer and preprocessing, feature schema, training dataset lineage, code and environment, evaluation reports, intended use, owners, approvals, deployment compatibility, and rollback target. The registry should represent lifecycle state and evidence, not just store blobs with version numbers.

Mechanism

For compound AI systems, prompts, retrieval indexes, policies, and judge versions may be just as material as weights. Record the bundle actually evaluated and deployed.

Direct answer

A feature store becomes useful when many models reuse governed features across offline training and low-latency serving, and teams need point-in-time correctness, ownership, freshness, lineage, and discovery. It is unnecessary overhead for a small batch model with simple versioned transformations. The decision depends on reuse and consistency cost, not organization fashion.

Mechanism

A feature store does not automatically prevent skew; producers, event-time semantics, backfills, defaults, and online materialization must share an enforceable contract.

Direct answer

Shadowing sends real inputs to a candidate without using its decisions, which tests compatibility and performance but not user impact. A canary exposes a bounded cohort to the candidate with automatic guardrails and rollback. Champion/challenger runs alternatives over time for comparison; it needs clear traffic assignment, label joins, and decision ownership.

Mechanism

Models may change user behavior and future training data, so release safety includes business and feedback-loop effects, not only service health.

Direct answer

Make every stage consume immutable versioned inputs and publish content-addressed or run-scoped outputs; separate orchestration metadata from artifacts; and define retry, timeout, ownership, and quality gates explicitly. A run should resume from validated checkpoints without silently overwriting an approved model, and promotion should be a separate audited action.

Mechanism

The pipeline is a state machine, not a long script. Backfills and retraining need concurrency controls, lineage, and a policy for late or corrected data.

Direct answer

Use batch inference when predictions can be computed ahead of demand and freshness allows scheduled updates; it improves throughput, cost, and operational simplicity. Use online inference when the decision depends on request-time context or low-latency freshness. Many systems combine precomputed candidates or features with a small online model.

Mechanism

The choice changes failure semantics: batch needs completeness, backfill, and publication guarantees; online needs tail-latency, admission, fallback, and availability controls.

Direct answer

Batching amortizes model and accelerator overhead across requests and raises utilization, but waiting to form a batch adds queueing latency and mixed sequence lengths can create wasted work or head-of-line effects. Dynamic or continuous batching admits work over time under maximum wait, token, and memory budgets. Tune against p50 and p99 latency, throughput, and fairness.

Mechanism

Request count is the wrong workload unit for variable-length models. Schedule by input and output tokens, memory, and expected decode work where possible.

Direct answer

I split latency into admission, queueing, routing, model load, prefill, decode, streaming, network, and downstream tool time, then slice by model, input/output tokens, batch, region, tenant, and fallback path. Long requests, cold loads, queue saturation, KV pressure, retry amplification, or provider cohorts often hide behind a healthy average. Mitigate with admission, separate pools, bounded lengths, scheduling, or fallback and verify tail behavior under realistic load.

Mechanism

Throughput and latency can move in opposite directions. A serving change that packs the GPU better may increase time-to-first-token for interactive requests.

Direct answer

Route from task, risk, latency, privacy, capability, cost, and current health—not a static model preference. Scale on queueing and workload units such as tokens or accelerator memory with warm capacity for cold-start-sensitive tiers. Fallbacks need evaluated quality contracts, compatible schemas, bounded retry ownership, and clear disclosure when capability changes.

Mechanism

A fallback is a product behavior, not merely infrastructure. It may need a smaller task, retrieval-only result, cached answer, human escalation, or explicit unavailability.

Direct answer

Quantization represents weights and sometimes activations with lower precision, reducing memory bandwidth and footprint so larger batches or models fit on the same hardware. Real speedups depend on kernels and shapes, and aggressive quantization can degrade quality unevenly. Re-evaluate target tasks, long-tail slices, calibration, throughput, latency, and hardware compatibility.

Mechanism

A smaller artifact is not proof of faster serving. End-to-end benchmarks must include dequantization, batching, KV cache, and the actual runtime.

Direct answer

Noisy or inconsistent labels cap learnable quality and can bias every metric. Class imbalance changes base rates and makes accuracy misleading; address it through collection, stratified or cost-aware sampling, class weighting, calibrated thresholds, and metrics focused on the rare outcome. Preserve the natural distribution in final evaluation unless the decision context explicitly differs.

Mechanism

Oversampling changes training exposure but does not change real prevalence. Probability estimates and thresholds may need correction and validation on representative data.

Direct answer

Define producers, types, units, ranges, null semantics, event time, freshness, and compatibility for every critical feature; validate at ingestion and before serving; and version transformations with the model. Additive schema changes may be compatible, but semantic changes require dual-read or dual-write, backfill, shadow comparison, and an explicit cutover.

Mechanism

A field can retain its type while changing meaning. Monitor distributions and provenance, not just schema shape.

Direct answer

Version datasets through immutable manifests over governed source records rather than unmanaged copies, with provenance, purpose, consent, retention, access, and deletion state. Privacy obligations may require excluding future runs, deleting derived artifacts, retraining, or documenting why exact removal is infeasible. The lineage graph must show which models and indexes depend on affected data.

Mechanism

Reproducibility and deletion can conflict. Design manifests, encryption, retention, and rebuild procedures together instead of treating privacy as a later metadata field.

Direct answer

I compare current and baseline populations, feature freshness and missingness, category vocabularies, upstream schema and join rates, label definitions and delays, feedback policy, and cohort mix. I replay known events through current and historical feature code to separate real-world concept change from a broken pipeline, then protect users with rollback, stale-safe features, or a rule fallback.

Mechanism

‘The model did not change’ only removes one variable. Data sources, transformations, routing, thresholds, and the population can all change the realized decision function.

Direct answer

Data drift means the input distribution changes; concept drift means the relationship between inputs and the target changes. Prediction drift is only an observable symptom and can result from either. I monitor important features and outputs by slice, but confirm business impact with labels or trusted outcome proxies before retraining automatically.

Mechanism

Not every drift is harmful, and harmful drift may occur in a small cohort hidden by global distributions. Alerts need baselines, seasonality, and an action owner.

Direct answer

Skew occurs when training and serving compute different values because code, dependencies, defaults, event-time joins, freshness, or missing-data behavior diverge. Reuse or compile one transformation definition where practical, log feature versions and sampled values, compare offline and online computation on the same entities and timestamps, and block incompatible schemas.

Mechanism

Shared code is helpful but insufficient if data availability differs. Point-in-time semantics and freshness contracts must match the real decision.

Direct answer

Monitor a hierarchy: service health; feature freshness and schema; input, score, and decision distributions; policy and override rates; user or operator proxies; and finally joined labels when they arrive. Calibrate proxies against eventual truth, sample high-risk decisions for review, and retain request-to-outcome identifiers so delayed evaluation is possible.

Mechanism

A proxy is an early-warning signal, not a quality target. Optimizing it can create a feedback loop or hide harm.

Direct answer

Correlate versioned data lineage, feature health, model inputs and scores, decision policy, service latency/errors, resource and cost telemetry, human overrides, safety events, and delayed business outcomes through one traceable prediction identity. Provide slice analysis, baseline comparisons, release annotations, ownership, and runbooks rather than one ‘model accuracy’ dashboard.

Mechanism

Telemetry must answer which cohort, version, and stage changed first. Sample and redact content deliberately to avoid turning observability into a privacy leak.

Direct answer

Classify quota, overload, timeout, and invalid-request failures; give one layer ownership of bounded exponential backoff with jitter; respect provider retry hints; and enforce deadlines, concurrency, and token budgets. Use circuit breakers and admission to stop retry amplification, then degrade to an evaluated smaller model, retrieval-only answer, cache, queue, or explicit unavailability.

Mechanism

Cross-provider fallback is not transparent if schemas, safety behavior, context limits, or data-processing terms differ. Every fallback needs tests and product semantics.

Direct answer

Preserve the highest-value deterministic function when probabilistic dependencies fail: show search results without synthesis, use a reviewed template, narrow the task, queue work, or request human review. State capability changes honestly, protect saved user work, and measure fallback quality. A silently weaker answer is not graceful degradation.

Mechanism

Design degradation from user outcomes and risk, then test it under provider, retrieval, tool, and telemetry failures.

Direct answer

Define user-journey SLOs across availability, end-to-end latency, answer or decision quality, safety, freshness, and cost, with separate indicators where quality labels arrive slowly. Segment by risk and capability tier, specify measurement windows and exclusions, and attach error-budget actions such as freezing releases, changing routes, or reducing autonomy.

Mechanism

Infrastructure availability is necessary but not sufficient: a fast fluent wrong answer can meet every conventional service SLO while the product fails.

Direct answer

Validate at every irreversible or trust-changing boundary: parse schema, resolve entities, enforce authorization and policy, check ranges and invariants, simulate or require approval for side effects, and verify postconditions. The model can propose and explain, but deterministic code owns contracts the business cannot probabilistically violate.

Mechanism

Validation after generation is only useful if failure has a safe recovery path. Design correction, escalation, and idempotency before enabling execution.

Direct answer

I decompose cost per successful task by model, tenant, route, input and output tokens, retrieval and reranking, tool calls, retries, agent steps, cache hits, and provider pricing. I compare the last good release and cohort mix, stop runaway loops or retry amplification, then optimize the largest measured term with quality gates and verify both unit economics and user outcomes.

Mechanism

Request volume can stay flat while context, generated length, failed attempts, or expensive routing grows. Cost per successful outcome is more useful than cost per request.

Direct answer

Define capability and risk tiers, route easy or low-impact tasks to smaller models, and escalate only when confidence, policy, or task complexity requires it. Evaluate the router and each path on the same outcome set, cap oscillation and retries, retain override and fallback rules, and monitor quality and spend by cohort.

Mechanism

A cheap classifier that misroutes hard cases can cost more through retries and failures. Routing is itself a model or policy with an evaluation and rollback lifecycle.

Direct answer

Semantic caching can reuse results for meaningfully equivalent low-risk requests, but similarity is not equality. Keys and eligibility must include tenant, identity or permission scope, locale, corpus and policy versions, model and prompt versions, freshness, and task-specific thresholds. Avoid it for personalized, rapidly changing, high-impact, or side-effecting requests unless correctness is provable.

Mechanism

Cache embeddings and immutable intermediate work more readily than final answers. Audit false hits as quality and isolation incidents.

Direct answer

Provide a shared gateway with identity, per-use-case attribution, budgets, model and region policy, caching and routing primitives, real-time unit-cost telemetry, and exception paths. Set default context and output limits, surface cost per successful product outcome, and review high-growth workloads with teams. Centralize leverage and evidence, not every product decision.

Mechanism

Chargeback alone can incentivize hiding usage or avoiding evaluation. Teams need forecasts, experiments, and paved-road optimizations before punitive controls.

Direct answer

Reproduce the first non-finite step and inspect the batch, inputs, targets, loss terms, activations, and gradient norms layer by layer. Common causes include invalid data, divide/log operations, excessive learning rate, exploding gradients, unstable custom loss, and mixed-precision overflow or underflow. Reduce the problem, fix the earliest invalid value, then restore scale and verify convergence.

Mechanism

Skipping the batch or lowering the learning rate may hide a deterministic data or numerical bug. Add finite-value assertions and retain the failing sample and run bundle.

Direct answer

Measure allocated, reserved, peak, and per-stage memory while reproducing the failing shape. Distinguish expected model/optimizer/activation or KV-cache demand from retained graphs, leaks, fragmentation, uneven sharding, and workload outliers. Then reduce batch or sequence, use appropriate precision, checkpointing, accumulation, sharding, bounded queues, or allocator tuning and verify throughput and quality.

Mechanism

Calling an empty-cache function does not free live tensors and is not a design. Memory controls must match the workload and runtime.

Direct answer

I align the release timeline with model, features, thresholds, routing, product UI, and policy changes; inspect outcome funnels and cohorts; compare score and decision distributions; join delayed labels; and replay representative traffic through old and new bundles. I look for feedback loops, calibration shifts, threshold or feature skew, and users changing behavior—not only service errors.

Mechanism

A technically healthy endpoint can serve systematically worse decisions. The incident boundary is the user journey, not the model process.

Direct answer

Start at the hardware and driver, then inspect device visibility, framework build, bundled CUDA runtime, environment selection, and loaded libraries in that order. Confirm the running interpreter and package, not the shell where installation happened. Reproduce in a minimal process before changing paths or reinstalling components.

Mechanism

System CUDA toolkits, drivers, and framework-bundled runtimes are related but not interchangeable. Random installation changes often create a second mismatch.

Direct answer

Clarify the surface, objective, feedback, freshness, scale, latency, and safety constraints. Separate candidate generation from coarse and fine ranking, then apply policy, diversity, and exploration in reranking. Define offline retrieval/ranking metrics, online outcome experiments, feature freshness, cold start, feedback loops, and degraded modes before optimizing models.

Mechanism

The system objective—not the algorithm name—drives architecture. Watch time, conversion, satisfaction, and safety can conflict and require constrained multi-objective decisions.

Direct answer

Define corpus, query types, freshness, permissions, relevance, traffic, and latency first. Build versioned ingestion into lexical, vector, metadata, and document stores; route authorized queries through hybrid retrieval, rank fusion, bounded reranking, and result assembly; then evaluate candidate recall, top-rank relevance, freshness, isolation, and tail latency independently.

Mechanism

Generation is optional. If users need evidence rather than synthesis, a high-quality search result page can be safer, faster, and easier to evaluate.

Direct answer

Start from decision latency, availability, freshness, correctness, and loss from false decisions. The path typically authenticates, fetches bounded fresh features, validates schema, invokes a versioned model, applies deterministic policy, records the decision, and returns inside a strict deadline. Add stale-safe defaults, circuit breakers, shadow/canary rollout, feature and score telemetry, and replayable logs.

Mechanism

The model may consume only a fraction of the budget. Feature fan-out, network, serialization, and policy often dominate p99 and reliability.

Direct answer

The gateway should provide workload identity, tenant policy, model abstraction without hiding material capability differences, quotas, routing, retries, streaming, redaction, audit, cost attribution, evaluation hooks, and versioned fallbacks. Keep prompts and product logic owned by teams while centralizing cross-cutting controls and evidence. Avoid a universal schema that erases provider-specific semantics needed for quality or safety.

Mechanism

A gateway becomes a critical control plane. Its data path must degrade safely, preserve backpressure, and avoid turning one provider outage into global retry amplification.

Direct answer

Build a platform when multiple teams repeatedly need the same hard capabilities—identity, model access, evaluation, observability, data controls, serving, cost attribution, and incident response—and shared ownership lowers total risk and time. The platform should offer paved roads and extension points, publish reliability and migration contracts, and measure adoption and developer outcomes rather than mandate one model or framework.

Mechanism

Centralize economies of scale and control-plane concerns, not domain data, product prompts, or every experiment. A platform without customer feedback becomes a bottleneck.

Direct answer

Treat user, retrieved, and tool content as untrusted data and assume text-only instruction hierarchy can be bypassed. Reduce impact with least-privilege tools, out-of-model authorization and policy, data/instruction separation, constrained schemas, output validation, approval for sensitive actions, isolation, budgets, and adversarial evaluation. Detection and filtering help but are not a complete boundary.

Mechanism

Prompt injection is dangerous when model interpretation can cross a privilege or data boundary. Minimize capabilities and verify every consequential action outside the model.

Direct answer

An attacker can add or alter content that ranks highly, embeds malicious instructions, impersonates authority, or exploits weak tenant filters. Defend with authenticated ingestion, source provenance and trust tiers, malware and content controls, authorization-aware indexing, anomaly and canary queries, instruction/data separation, citation validation, and rapid tombstone plus cache invalidation.

Mechanism

The risk spans ingestion, ranking, generation, and execution. A clean model cannot compensate for an untrusted corpus with broad tool access.

Direct answer

Issue scoped workload identity per user, tenant, and task; authorize each typed operation outside the model; separate read, draft, and execute capabilities; validate arguments and targets; require approvals for high-impact actions; sandbox untrusted computation; make side effects idempotent; and retain tamper-resistant audit records. The model never receives reusable broad credentials.

Mechanism

Tool descriptions influence selection but do not enforce policy. The executor must re-derive identity and scope from trusted state, not model-provided fields.

Direct answer

Derive tenant and principal identity from verified credentials, propagate it through retrieval, caches, prompts, tools, logs, and model calls, and enforce authorization before candidate selection. Bind every derived artifact and cache key to tenant, permissions, corpus snapshot, and relevant versions; partition infrastructure when risk demands it; and continuously test cross-tenant canaries and deletion.

Mechanism

Filtering final results is too late: unauthorized documents can influence ranking, generation, caches, and telemetry even if citations are removed later.

Direct answer

Start from the decision, affected groups, legal and product context, and specific harms; then examine data representation, label quality, error and calibration differences, allocation or exposure outcomes, and intersections. Mitigation may change collection, labels, objectives, thresholds, product policy, review, or whether automation is appropriate. Document trade-offs because fairness criteria can conflict.

Mechanism

A global parity metric can hide subgroup harm and cannot choose the normative goal. Include domain, policy, legal, and affected-user expertise in the decision.

Direct answer

Explanations are useful when they support debugging, contestability, operator decisions, or required disclosure, but they must match the audience and cannot be treated as proof of causal truth. Human oversight helps only when reviewers have authority, time, context, training, and escalation paths; otherwise it becomes ceremonial automation bias.

Mechanism

Design the action the explanation enables. For high-impact decisions, preserve evidence, offer appeal, and measure reviewer overrides and outcomes.

Direct answer

Create risk tiers tied to required evidence and controls, assign product and technical owners, provide shared evaluation, registry, policy, observability, and incident tooling, and keep an exception process with expiry and accountable approval. Review data, models, vendors, safety, security, privacy, and user impact across the lifecycle while preserving team autonomy for low-risk experimentation.

Mechanism

Governance should make safe delivery easier through paved roads and visible evidence. Committees without integrated engineering controls create paperwork after decisions are already made.