Design reviewAdvanced95 min estimateOriginal publication

Prompt, Retrieve, Fine-Tune, or Train?

Diagnose the missing resource before purchasing a more expensive way to change the system.

Authorship: InterviewsVector
Published / updated: 2026-09-19 / 2026-09-19
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable illustrative contracts are covered by focused tests and primary sources are recorded. No named human review or empirical model improvement is claimed.

The decision in one pass

Choose the smallest intervention that passes a qualified evaluation under the actual product constraints. First distinguish missing knowledge from missing capability, unreliable format, excessive latency, privacy restrictions, and stale information. A clearer prompt may expose an existing capability; retrieval may supply current, attributable knowledge; supervised fine-tuning may make a recurring behavior more reliable; changing or training the base model may address a deeper representation or capability gap. These are hypotheses to test, not an automatic maturity ladder. If prompting or retrieval already meets the bound quality, freshness, privacy, latency, cost, and risk requirements, the evidence does not justify training. Missing baseline evidence means investigate, not escalate.

Why this matters

Training creates a new model release, data obligation, evaluation burden, and rollback problem. It cannot repair every retrieval outage, unclear instruction, or permission error. A decision record makes the causal deficit and the evidence for each rejected alternative inspectable before compute and operational complexity accumulate.

You will be able to

Classify failures by knowledge, capability, format, latency, privacy, and freshness instead of treating every miss as a training problem.
Construct comparable, held-out prompt and retrieval baselines before escalating intervention size.
Bind evaluation evidence to model, data, policy, scope, version, and content identity.
Apply owner, rollback, cost, risk, and operational constraints as release gates rather than footnotes.
Explain when changing the base model is justified and when evidence is merely incomplete.

Prerequisite contract

Your Vector Loop for this lab

01
Model
State the user outcome, failure cohort, deficit hypothesis, scope, and binding product constraints.
02
Derive
Derive what each intervention can change and the evidence needed to reject cheaper alternatives.
03
Build
Produce a content-addressed decision record with qualified comparative evaluations and an accountable owner.
04
Stress
Remove baseline evidence, change model versions, violate privacy, and test stale or unrepresentative evaluations.
05
Operate
Shadow the chosen intervention, monitor cohort regressions, and retain a tested rollback route.
06
Defend
Explain why this intervention is sufficient for this scope and what further evidence would reverse the choice.

A failure is evidence about a missing resource

Start with a failing task and a counterfactual. If the model answers correctly when supplied the right source, the immediate deficit is likely knowledge access, not an inability to perform the task. If it still fails with a verified source and clear instructions, investigate the reasoning, representation, tool, or task capability. If it produces the right content in the wrong schema, diagnose output control separately. This decomposition gives experiments a purpose: change one resource while preserving the other conditions.

Deficit	Discriminating probe	Candidate response
Knowledge	Provide a verified relevant passage	Retrieve and cite authorized evidence
Capability	Keep evidence complete and inspect systematic task failures	Compare a stronger base, then evaluate adaptation
Format	Use clear schema, examples, and validation	Prompt or constrained output; SFT only if persistent
Latency	Decompose retrieval, prefill, decode, and queue time	Optimize the measured component; do not assume training helps
Privacy	Trace every data boundary and retention rule	Restrict deployment/data access and reevaluate
Freshness	Replace stale evidence with current authorized material	Use an updateable source and freshness gate

Compare what changes, not what sounds more advanced

Prompting changes the inference-time specification and examples. Retrieval changes the information available to the model at inference, with an explicit source and update path. Fine-tuning changes parameters to alter repeated behavior under an objective. Training or replacing the base changes the representation and capability starting point, usually with a much larger evidence and operational burden. A composition can be appropriate: a tuned assistant can still retrieve current policy documents. The decision is about the smallest sufficient product change, not loyalty to one technique.

Knowledge in parameters is difficult to edit precisely and does not inherently carry a citation or access-control check. Retrieval introduces its own failure modes: missing documents, bad ranking, stale indexes, malicious source text, and excessive latency. SFT can improve adherence to demonstrated patterns while reinforcing annotation errors or overfitting repeated answers. These are different control surfaces. Name the mechanism you expect to improve and the regression surface you will measure.

choose the first qualified candidate satisfying quality ∧ cost ∧ risk ∧ latency ∧ privacy ∧ freshness

The teaching policy orders prompt, retrieval, fine-tune, and train. It is a declared preference for intervention size, not a proof of universal total-cost optimality. A production team must document its own ordering and comparable baselines.

A comparison must be bound to the same question

Freeze the task scope, base model revision, evaluation data, rubric, access policy, and release version. Keep evaluation cases outside prompt-example selection and training. Record actual output artifacts and their content digests, not only a dashboard URL. Minimum case counts are necessary but not sufficient: stratify high-risk and long-tail cohorts, retain uncertainty, and inspect failures. The fixture checks declared qualification and bindings; it cannot independently certify a human's claims about representativeness or the external evaluation run.

01Measure the existing systemEstablish a reproducible baseline and map errors to deficit hypotheses. Include refusals, permission-denied cases, and current-document tasks.
02Test an inference-only interventionImprove instructions and source access while holding the evaluation protocol constant. Record unresolved errors and operational constraints.
03Escalate only on qualified failuresTry a training-based candidate only after smaller candidates fail the declared requirements. Compare with the same acceptance evidence, not a hand-picked demo.

Make the decision auditable and immutable

The artifact uses frozen concrete records, defensive copies, unique evidence identities, public-boundary revalidation, finite bounded numeric fields, and canonical content hashing. Its illustrative freshness case passes retrieval after the prompt baseline fails. The report deliberately says that sufficiency holds in this declared order and scope. The digest binds costs, risk, thresholds, provenance, owner, rollback, and external evaluation content references; changing any bound field creates a different record.

adaptation_decision_audit.py

1def main():
2    result = audit(example_contract())
3    print("example=illustrative_only")
4    print("intervention=" + result.intervention)
5    print("claim=" + result.claim)
6    print("training_required=" + str(result.intervention in ("fine-tune", "train")).lower())

Expected output

example=illustrative_only
intervention=retrieval
claim=SMALLEST_SUFFICIENT_IN_THIS_ORDER
training_required=false

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/adaptation_decision

Do not confuse immutability with trust. A forged frozen object can be mutated through low-level Python operations; audit reconstructs and validates records at its public boundary. Likewise a 64-character digest binds an external artifact reference but does not fetch or judge that artifact. Qualification of the referenced evaluation remains an upstream responsibility. The tests prove local contract behavior, not a real model's performance.

Make the chosen intervention reversible

A decision record needs an owner with authority to stop the rollout, a known-good release, and a rollback path that restores model, prompt, retrieval index, access policy, and evaluator compatibility together. Shadow before exposing users, canary by meaningful cohort, and observe task success alongside latency, retrieval coverage, source age, privacy denials, and cost. A lower average error rate does not authorize degradation on a protected or high-risk cohort.

Specify reversal triggers before rollout. A retrieval release may be reversed after stale-source incidents; an SFT release after behavior regressions or newly discovered data-rights issues. The decision also expires when its premises change: a new base model, product scope, freshness requirement, source corpus, or deployment boundary invalidates old comparisons. Reusing a friendly version name must not preserve the old content identity.

Operate at three altitudes

Production lens

— Record prompt, model, source-index, data-policy, and evaluator versions on every comparative run; reject cross-scope scores.
— Monitor freshness, permission failures, tail latency, cost, and task quality together, including cases where the system should abstain.
— Keep a named rollback owner and restore the complete system configuration, not only a weight file.

Staff lens

— Require a falsifiable deficit hypothesis and qualified cheaper-baseline evidence before authorizing a training program.
— Compare lifecycle cost: annotation, evaluation, operations, compliance, on-call load, and future migrations, not only one training invoice.
— Define who can waive constraints and make waivers explicit; the default artifact does not silently waive any gate.

Interview defense

Your support assistant gives wrong answers. How do you decide whether to prompt, retrieve, fine-tune, or train?

I first separate missing knowledge, capability, format, freshness, latency, and privacy constraints. I reproduce failures on held-out cohorts, then test whether a clearer specification or verified source repairs them. I compare qualified candidates under the same scope, model/data versions, rubric, cost, risk, and operational gates. If prompting or retrieval is sufficient I stop; missing baseline evidence blocks escalation. A behavior gap can justify SFT, while deeper capability gaps require stronger evidence and possibly a different base. I bind the decision to artifacts, name an owner, and define rollback and expiry conditions.

Expect the interviewer to press on

— What would disprove your knowledge-deficit hypothesis?
— Why is a missing baseline different from a failed baseline?
— When must this decision be reevaluated?

Misconceptions to remove

“Fine-tuning is the next step once prompting exists.”

It is a different intervention. Escalate only when qualified smaller interventions fail the scoped requirements.

“Training is the best way to keep facts current.”

Mutable sources with explicit freshness and access controls often address current knowledge more directly; test retrieval first.

“A single accuracy score settles the architecture.”

Quality is one gate alongside cost, risk, privacy, latency, freshness, operational ownership, and reversibility.

Check your model

1. The retrieval candidate passes every gate but training scores slightly higher. What does the declared ladder select?

Retrieval: it is the smallest sufficient candidate under the bound policy. Extra score alone does not establish a requirement for training.

2. What happens if prompt evidence is absent?

The decision blocks. An untested smaller candidate has not been shown insufficient.

3. Why bind source data and policy to evaluation evidence?

A score measured with different documents, permissions, or scope does not qualify the candidate being released.

Prove the mechanism

Create six illustrative failure cohorts, one per deficit. Define a discriminating probe and the smallest candidate that could fix each failure. Add tests that prevent an attractive training score from bypassing missing prompt or retrieval evidence.

Add a production constraint

Write a rollout and reversal plan for an assistant whose policy documents change daily but whose output format is stable. Separate retrieval freshness, format adherence, privacy, and latency evidence; specify when a mixed retrieval-plus-SFT design becomes justified.

Artifact: Adaptation decision record

courses/ai-engineering/reference-impl/adaptation_decision/adaptation_decision_audit.py

Download reference implementation

Primary references and next links

References

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al.. Primary work combining parametric models with explicit retrieved knowledge.
2. Finetuned Language Models Are Zero-Shot Learners
Wei et al.. Primary instruction-tuning work; motivates testing behavior adaptation rather than promising universal gains.
3. LoRA: Low-Rank Adaptation of Large Language Models
Hu et al.. Primary low-rank adaptation method, one option after training has been justified.

Continue through the graph

Supervised Fine-Tuning as Behavior Shaping →
Carry the justified behavior gap into a dataset contract.
LoRA and the Low-Rank Update Hypothesis →
Budget one possible parameter-efficient intervention without assuming sufficiency.
AI/ML interview questions →
Practice defending the intervention choice and its limits.

Glossary: adaptation · knowledge deficit · capability deficit · held-out evaluation · content digest · rollback · retrieval-augmented generation