Concept labAdvanced105 min estimateOriginal publication

Pretraining Objectives Shape Model Behavior

An objective defines which errors receive credit. It does not, by itself, prove why a capability appeared or whether the deployed behavior is safe.

Authorship: InterviewsVector
Published / updated: 2026-08-25 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

A training objective couples a prediction target, label construction, loss mask, reduction, data distribution, tokenizer, model, and optimization process. Causal language modeling scores the next token given the prefix; masked language modeling scores selected hidden tokens with bidirectional context; text-to-text mixtures express several tasks through token prediction; preference stages optimize evidence derived from comparisons under still more assumptions. These choices change the learning signal, but an observed capability is jointly shaped by architecture, scale, data contents and mixture weights, optimization, evaluation, and post-training. Audit token IDs, shifted labels, active masks, per-token losses, objective and tokenizer versions, and the exact data-mixture digest. Report alignment and supported loss only—never a causal capability claim from one batch.

Why this matters

Teams often describe a model as trained on next-token prediction as if that phrase explains every behavior. In practice, a one-position label shift, stale tokenizer, hidden padding loss, duplicated source mixture, or changed reduction can alter the training signal long before aggregate loss looks suspicious. Evidence-bound objective audits make the actual supervision inspectable.

You will be able to

Distinguish causal, masked-token, text-to-text, and preference-derived supervision.
Derive causal token/label shifting and active-loss masking.
Bind objective, tokenizer, model, data-mixture, example, and reduction identities.
Compute a stable supported token mean without including ignored positions.
Separate objective alignment evidence from downstream capability attribution.
Design data-mixture, contamination, slice, and post-training evaluation gates.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Name prediction target, context visibility, label construction, mask, reduction, tokenizer, data mixture, and training stage.
02
Derive
Derive which token positions receive loss and how per-token values become a reported aggregate.
03
Build
Capture immutable batch evidence and verify causal shift, ignore sentinel, active support, scope, and content digests.
04
Stress
Inject off-by-one labels, padding loss, stale tokenizers, forged mixtures, duplicate documents, sparse support, and numeric attacks.
05
Operate
Trace objective and data lineage through training, checkpoint evaluation, post-training, and deployment cohorts.
06
Defend
State what alignment and loss evidence show, which confounders remain, and why capability attribution needs controlled comparisons.

Treat the objective as a supervision contract

A loss formula is only the center of the contract. The complete surface includes the source documents, mixture weights, tokenizer, example packing, corruption or masking procedure, labels, ignore sentinel, reduction denominator, model revision, optimizer, and training stage. Two runs that both say cross entropy may train on different targets.

The objective specifies a learning signal; none of these rows alone guarantees a downstream capability.
Objective family	Visible context	Scored target	Boundary
causal LM	prefix	next token	cannot condition on future tokens
masked LM	declared left and right context	selected hidden tokens	mask/corruption process changes input
text-to-text mixture	task-formatted input	target sequence	mixture and template define supervision
preference-derived stage	prompt and candidate evidence	relative or policy signal	comparison and reference policy assumptions

Derive causal labels one position at a time

L = −(1 / |A|) Σᵢ∈A log pθ(xᵢ₊₁ | x≤ᵢ)

A is the declared set of active positions. The token at i+1 is the label for position i; the final position or padding may use the declared ignore sentinel and must not enter the denominator.

01Freeze tokenizationBind tokenizer identity and version before constructing labels; the same text under another tokenizer is different evidence.
02Shift exactly onceAt each active position i, require label[i] = token[i+1]. A second shift or no shift trains another task.
03Mask explicitlyInactive positions carry the declared sentinel and exactly zero recorded token loss in this audit convention.
04Reduce over supportCount active tokens, use an accurate sum, and refuse batches below the declared minimum support.

Make the data mixture part of the objective identity

Sampling weights decide how often each source contributes gradient evidence. Filtering, deduplication, language balance, quality scoring, contamination removal, and example packing all change the effective distribution. A friendly mixture name is insufficient: bind a version and content digest, then preserve source partition and document identity in each audited example.

Change	Aggregate loss may	Evidence needed
more duplicated easy text	improve	dedup and document lineage
new tokenizer	move abruptly	old/new retokenized replay
padding enters loss	look artificially easy	active-mask denominator
mixture reweighting	hide slice regressions	partition-level loss and support
benchmark contamination	improve misleadingly	time-aware source and overlap audit

Audit the batch without claiming capability

training_objective_audit.py

1def audit_training_objective(
2    contract: ObjectiveContract, evidence: ObjectiveEvidence
3) -> ObjectiveAuditReport:
4    """Validate objective alignment without attributing downstream capability."""

Expected output

example=illustrative_only
contract_version=training-objective-audit-v1
objective=causal-next-token-cross-entropy
evidence_content_id=objective-evidence@sha256:2a8ef76110399fe1a592ea5d0f3ff81488052174adb179a3f0175143bade4378
active_tokens=3
mean_token_loss=0.600
status=ALIGNED
claim=OBJECTIVE_ALIGNMENT_ONLY_NO_CAUSAL_CAPABILITY_CLAIM

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/training_objective

The excerpt is literal artifact source. The audit binds objective, label policy, mask policy, tokenizer, data-mixture ID/version/digest, model, trainer source, example, document, and owner. It revalidates concrete frozen tuples, refuses duplicate examples and documents, enforces exact next-token labels and ignored positions, caps examples and tokens, rejects booleans, NaN, infinity, sub-resolution or excessive loss, and recomputes every content identity.

Stress lineage, support, and label construction

01Shift one label incorrectlyRefuse the batch even if the recorded loss is finite and plausible.
02Score an ignored positionRequire the declared sentinel and zero recorded loss for every inactive token.
03Reuse a documentReject duplicate document identity so support cannot be manufactured by copying evidence.
04Change only the mixture digestInvalidate the batch scope even when objective and tokenizer names remain identical.
05Attack the numeric boundaryReject bools, NaN, infinity, sub-resolution active loss, excessive loss, and unsupported aggregate arithmetic.

Operate objectives through stage-aware evaluation

Stage	Primary evidence	Do not collapse into
pretraining	token loss by source and shape	assistant helpfulness
instruction tuning	task and template slices	base-model likelihood
preference stage	comparison quality and policy shift	ground-truth factuality
deployment	task, safety, latency, and feedback metrics	training loss

Version each stage and retain the input checkpoint, data contract, objective implementation, tokenizer, optimizer, seed policy, and evaluation card. Compare controlled checkpoints when asking causal questions. A pre/post result without a matched baseline can suggest a hypothesis, but it cannot isolate objective, data, or optimizer effects.

Use controlled comparisons for capability claims

To argue that an objective change affected a capability, hold architecture, data, tokenizer, compute, optimizer, and evaluation fixed where possible; vary the objective; run enough seeds or uncertainty analysis; and inspect negative as well as positive task slices. Even then, state the experimental scope. The artifact in this lesson verifies training-record alignment only.

Operate at three altitudes

Production lens

— Track active-token counts and loss by data partition, language, sequence length, and training stage.
— Bind tokenizer, label, mask, reduction, objective, data-mixture, model, and trainer revisions in every replay.
— Audit duplicate documents, contamination windows, source licenses, and mixture-weight changes separately from loss.
— Evaluate downstream task, safety, calibration, and robustness after every objective or data-stage change.

Staff lens

— Require objective changes to state the intended learning signal and the evaluation capable of falsifying it.
— Separate alignment of the training record from attribution of downstream behavior.
— Fund data lineage and controlled checkpoint comparisons as first-class model infrastructure.

Interview defense

How do pretraining objectives affect model behavior, and how would you audit one?

An objective defines targets, context visibility, masks, reduction, and how a data mixture supplies gradient evidence. Causal LM predicts the next token from a prefix; masked LM predicts selected hidden tokens with broader context; text-to-text and preference stages encode other supervision through their formatting and evidence. I would bind tokenizer, objective, label and mask policies, data-mixture digest, model, and trainer versions; verify per-position labels and active support; and report slice loss. I would not infer a capability from the objective or one batch—causal attribution needs controlled checkpoints and downstream evaluation.

Expect the interviewer to press on

— How can padding lower the reported loss incorrectly?
— What would you hold fixed to compare two objectives?
— Why is a mixture name insufficient provenance?

Misconceptions to remove

“Next-token prediction directly teaches every observed capability.”

The objective supplies one learning signal; behavior is jointly shaped by data, architecture, scale, optimization, post-training, prompting, and evaluation.

“Cross-entropy loss is comparable across tokenizers and masks.”

Token units, active positions, label construction, and denominators change the measurement contract.

“A valid average loss proves the data pipeline is correct.”

Duplicates, contamination, stale mixtures, and mislabeled slices can produce plausible aggregates; lineage and slice evidence remain necessary.

Check your model

1. For causal LM tokens [a, b, c], which labels are active under a final-position ignore convention?

The positions for a and b use labels b and c. The final c position is inactive and carries the declared ignore sentinel.

2. Why must a data-mixture content digest be bound even when its friendly version name is unchanged?

The actual source contents or weights can change under a reused name; the digest makes that semantic drift observable.

3. What does an ALIGNED batch report not establish?

It does not establish representativeness, data quality, downstream capability, causality, safety, or production usefulness.

Prove the mechanism

Add a separately versioned masked-token objective contract with deterministic corruption evidence. Compare its active positions and visibility rules to the causal audit without reusing either claim boundary.

Add a production constraint

Design a controlled objective comparison with fixed data, tokenizer, architecture, compute, optimizer budget, and blinded evaluation. Specify which residual confounders and uncertainty remain.

Artifact: Training objective audit

courses/ai-engineering/reference-impl/training_objective/training_objective_audit.py

Download reference implementation

Primary references and next links

References

1. Language Models are Few-Shot Learners
Brown et al.. Primary autoregressive language-model paper and downstream evaluation reference.
2. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin et al.. Primary masked-language-modeling paper used to contrast visibility and target construction.
3. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al.. Primary systematic study of objectives, data, architectures, and transfer settings.

Continue through the graph

The Generalization Contract →
Separate transfer and evaluation evidence from training loss.
Decoding Is a Product Policy →
Distinguish the training distribution from runtime token selection.

Glossary: causal language modeling · masked language modeling · label shift · loss mask · data mixture · active-token mean