Failure labAdvanced105 min estimateOriginal publication

Position, Context, and Extrapolation

Position mechanisms impose different order and distance structures; algebraic support beyond training length is not evidence of useful long-context behavior.

Authorship: InterviewsVector
Published / updated: 2026-08-25 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Self-attention needs positional structure because content-only routing is permutation-equivariant. Absolute embeddings add a position-specific vector; sinusoidal features encode positions with fixed frequencies; RoPE rotates query/key coordinate pairs so their dot product exposes relative offset; ALiBi adds a head-specific distance penalty to attention logits. Compare these mechanisms by the invariant and intervention they implement, then bind training range, declared serving context, offsets, scaling rules, precision, and model revision. A formula that evaluates at a longer index—or a relative-shift invariant that still passes there—does not prove the model learned to use that length. Long-context claims require task evidence across positions and failure modes.

Why this matters

Extending a context configuration can produce valid tensors and plausible text while silently degrading retrieval, order tracking, calibration, latency, or memory. Position policy is therefore both a model-behavior contract and a serving-capacity decision.

You will be able to

Explain why content-only self-attention cannot distinguish permutations without positional information.
Compare learned absolute, sinusoidal, RoPE, and ALiBi mechanisms by where position enters and which invariants they encode.
Derive RoPE's relative dot-product shift invariant for paired coordinates.
Separate training context, declared serving context, representable index, and empirically validated context.
Build a content-addressed audit that tests one bounded invariant while refusing an unsupported quality claim.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Name order requirements, positional mechanism, index origin, training range, serving range, precision, and quality claim.
02
Derive
Derive where positional information enters and the exact invariant each mechanism is expected to preserve.
03
Build
Capture bounded vectors and positions, then test a content-addressed RoPE relative-shift invariant.
04
Stress
Probe offset errors, boundary positions, long distances, scaling changes, precision, distractors, and lost-in-the-middle behavior.
05
Operate
Monitor quality by position and length alongside cache, latency, truncation, and model/config identities.
06
Defend
Separate mathematical definability, invariant preservation, task accuracy, and a supported production context.

Add order because content routing alone cannot see it

Without a position-dependent signal, permuting token representations permutes self-attention outputs in the same way. The mechanism can compare content but has no independent basis for ‘first,’ ‘previous,’ or a distance of twelve. Position information deliberately breaks or structures that symmetry.

Attention(PX) = P·Attention(X) when position contributes no separate signal

P is a permutation of sequence rows. Content-only self-attention is permutation-equivariant: it follows the reordered rows rather than recovering their original order.

Compare mechanisms by where position enters

These mechanisms are not interchangeable checkpoint decorations; each changes a different boundary of the attention computation.
Mechanism	Intervention	Useful invariant or bias	Boundary risk
learned absolute	add learned vector by index	distinct trained index identity	untrained or unavailable rows
sinusoidal	add fixed multi-frequency vector	deterministic features at arbitrary indices	model may not use unseen phases well
RoPE	rotate Q/K coordinate pairs	dot product depends on relative offset	frequency/scale and long-distance degradation
ALiBi	add slope × distance to logits	explicit distance penalty	task and head behavior beyond training range

An invariant states what a transformation preserves. It is narrower and more testable than saying one encoding ‘understands order.’ Learned absolute vectors privilege specific indices. Sinusoids provide structured deterministic features. RoPE preserves a relative-shift relationship in query-key dot products. ALiBi imposes a monotonic distance bias. None of these statements predicts task quality by itself.

Derive the RoPE relative-shift invariant

⟨R(p)q, R(r)k⟩ = ⟨q, R(r−p)k⟩

R rotates each coordinate pair by a position-dependent angle. Because planar rotations compose by angle addition, shifting p and r by the same amount leaves their relative angle—and therefore this dot product—unchanged.

Different coordinate pairs use different frequencies, typically derived from a base and pair index. The implementation must bind dimension pairing, base, position offset, and any scaling or interpolation rule. A pair order swap or off-by-one position can preserve shapes but violate the intended relationship.

01Choose a bounded q, k, p, and rUse non-symmetric vectors and positions with a nonzero relative offset so indexing defects cannot hide behind zeros.
02Apply one common translationShift both positions without changing q or k and remain inside the declared serving range.
03Compare the two rotated dot productsUse a declared absolute tolerance and record whether shifted positions cross the training boundary.

Keep four different context bounds separate

Bound	Meaning	What it does not prove
representable index	the formula or table can produce a value	the checkpoint saw or uses it
training context	positions and dependencies exposed during training	uniform competence up to that limit
declared serving context	runtime admission and memory policy	acceptable task quality
validated context	range supported by chosen evaluations and thresholds	all tasks or future traffic

A context-length release should record each bound explicitly. Evaluation must stratify evidence location and dependency distance: beginning, middle, end, short hops, long hops, multiple distractors, repeated keys, and generation beyond the prompt. Otherwise a single aggregate score can hide positional collapse.

Test one invariant without laundering it into quality

The reference artifact binds mechanism, invariant, model and layer revision, training and declared context lengths, offset, even coordinate dimension, RoPE base, shift bound, numeric policy, provenance, scope, and owners. It reconstructs concrete frozen cases at the public boundary and content-addresses the material result. The illustrative shift crosses the training boundary; the invariant passes, while the extrapolation conclusion remains quality unproven.

position_extrapolation_audit.py

1def format_example() -> str:
2    report = audit_position_extrapolation(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_EVIDENCE)
3    case = report.cases[0]
4    return "\n".join(
5        (
6            "example=illustrative_only",
7            f"contract_id={report.contract_content_id}",
8            f"evidence_id={report.evidence_content_id}",
9            f"mechanism={report.mechanism}",
10            f"relative_offsets={case.relative_offset},{case.translated_relative_offset}",
11            f"maximum_position={report.maximum_observed_position}",
12            f"training_range={report.training_range_status}",
13            f"invariant={report.invariant_status}",
14            f"extrapolation={report.extrapolation_conclusion}",
15            f"material_id={report.material_evidence_id}",
16            f"decision={report.decision}",
17        )
18    )

Expected output

example=illustrative_only
contract_id=position-contract@sha256:12ab45b241be5d9570217884653ee1362f310c1e9737738876423c33dfa27e47
evidence_id=position-evidence@sha256:573ed990ca126ce89d76981a0723ce2ca916d30d63eb4109122ff51e6540a315
mechanism=rope-complex-pair-rotation-v1
relative_offsets=1,1
maximum_position=5
training_range=OUTSIDE_TRAINING_RANGE
invariant=PASS
extrapolation=INVARIANT_PASSED_QUALITY_UNPROVEN
material_id=position-material@sha256:45bd25d7fb7099360f55f7037e3eff7e0c0ff3cee30eebbd81fe1b85fadacc27
decision=PASS

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/position_extrapolation

Evaluate long context as a positional reliability curve

Evaluation axis	Example probe	Operational signal
absolute location	same evidence near start, middle, end	accuracy by normalized position
dependency distance	query and evidence separated by growing gaps	quality versus distance
distractor density	repeated or semantically similar decoys	precision and false retrieval
prompt/generation split	long prefill then long decode	quality, latency, and cache growth
position intervention	base/scale/offset variants	regression by exact policy identity

Gate a context extension with a workload distribution, not one synthetic needle. Include tasks whose answer requires combining distant evidence, track abstention and unsupported claims, and compare short-context controls so a gain at long length does not conceal a regression on ordinary requests. Capacity planning must run beside quality evaluation because cache memory and prefill cost grow with admitted length.

Operate at three altitudes

Production lens

— Bind position mechanism, base or slopes, scaling rule, index origin, packing offsets, model revision, context bounds, precision, and cache identity.
— Evaluate quality by evidence position, dependency distance, distractor density, prompt length, and generated length—not one maximum-context score.
— Pair long-context quality gates with prefill latency, KV-cache memory, admission, truncation, and concurrency evidence.
— Invalidate or version cached state when any position interpretation changes, even if tensor shapes remain load-compatible.

Staff lens

— Define representable, training, serving, and validated context as separate owned fields across model cards, gateways, SDKs, and dashboards.
— Require every context-extension proposal to state the positional intervention, checkpoint assumptions, evaluation matrix, capacity impact, and rollback path.
— Build a shared positional reliability suite that controls evidence location and distance while preserving privacy and reproducibility.

Interview defense

Compare positional mechanisms and explain why evaluating at a longer index does not prove context extrapolation.

Content-only self-attention is permutation-equivariant, so position must enter the function. Learned absolute embeddings add index-specific vectors; sinusoids add fixed frequency features; RoPE rotates Q/K pairs so their dot product depends on relative offset; ALiBi adds a distance penalty to logits. I would bind mechanism, offsets, scales, model revision, training range, serving range, and precision, then test its algebraic invariant. But definability or an invariant beyond the training range is not task competence. I would evaluate retrieval and composition across positions, distances, distractors, and generation lengths while measuring latency and KV-cache capacity.

Expect the interviewer to press on

— What does RoPE preserve under a common position shift?
— Why are representable and validated context lengths different?
— What must be invalidated when the position policy changes?

Misconceptions to remove

“If a positional formula accepts index 100,000, the model supports 100,000 tokens.”

Mathematical definability says nothing about learned use, task accuracy, memory capacity, or safe serving at that index.

“RoPE is purely relative because its attention dot product depends on relative offset.”

RoPE applies absolute-position rotations to Q and K; a relative relationship emerges in their dot product. Implementation still depends on absolute index origin, base, pairing, and scaling.

“Passing a synthetic needle-in-a-haystack task proves long-context reasoning.”

It tests one retrieval pattern. Robust evidence varies location, distance, distractors, composition, generation, and real workload structure with explicit thresholds.

Check your model

1. Why does content-only self-attention need positional information?

Without an independent position signal, permuting input rows only permutes outputs, so the operation cannot distinguish original order or distance from content alone.

2. What does the tested RoPE shift invariant establish?

It establishes that the bound rotation implementation preserved the query-key dot product when both positions shifted equally within the declared numeric tolerance. It does not establish long-context quality.

3. Which context length should an API advertise?

A serving limit supported by explicit model-quality and capacity evidence for the intended workload, labeled separately from training and merely representable bounds.

Prove the mechanism

Add within-training and outside-training cases at several relative offsets. Report invariant error and range status separately, then design one task evaluation that could test quality rather than algebra.

Add a production constraint

Write a context-extension release memo comparing unchanged RoPE, one scaling intervention, and ALiBi-style bias. Include compatibility, evaluation slices, cache/capacity impact, rollout, and rollback.

Artifact: Position extrapolation audit

courses/ai-engineering/reference-impl/position_extrapolation/position_extrapolation_audit.py

Download reference implementation

Primary references and next links

References

1. Attention Is All You Need
Vaswani et al.. Primary source for the Transformer's sinusoidal positional encoding.
2. RoFormer: Enhanced Transformer with Rotary Position Embedding
Su et al.. Primary source proposing RoPE and analyzing its relative-position structure.
3. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, Smith, and Lewis. Primary paper proposing ALiBi and evaluating train-short/test-long behavior.

Continue through the graph

Derive Attention from Content-Based Routing →
Locate where position modifies attention routing.
KV-cache capacity planning →
Connect context policy to persistent serving memory.
Matrices as Small Programs →
Review rotations and periodic coordinate features.

Glossary: positional encoding · permutation equivariance · rotary position embedding · relative position · context extrapolation · validated context