Derive Attention from Content-Based Routing

For every output position, a query vector expresses a learned request. Each candidate source position offers a key used for compatibility and a value used as payload. Query-key similarity chooses a routing mixture; it is the values—not the keys—that are combined into the context. This vocabulary exposes common bugs more clearly than saying a token simply ‘looks at’ another token.

Axis names are stronger than raw dimensions. A square score matrix can survive an accidental transpose, and equal Q/K/V widths can hide a semantic swap. Tests should use unequal query and key counts when possible, retain position IDs, and include a fixture whose expected direction is obvious.

1. 01Project in one model revisionBind Q, K, and V projection contents, layer/head identity, source sequence, and feature semantics.
2. 02Score only matching key dimensionsReject broadcasting or reshaping that changes which coordinates form each dot product.
3. 03Route values over the key axisThe score row length and value row count must match exactly before weighted summation.

If independent query and key coordinates have roughly unit variance, their dot-product variance grows with dk. Dividing by √dk controls that scale before softmax. It is not an arbitrary temperature knob: changing or omitting it changes the bound attention function and should change the contract identity.

A production kernel may tile the computation and avoid materializing the full score matrix, yet exact implementations must preserve the same masking, scaling, and normalization semantics. Compare outputs within a declared numeric envelope and retain the kernel, dtype, and accumulation policy alongside evidence.

The laboratory uses a bounded two-dimensional fixture. Apply position visibility, compare the scaled query-key scores, and predict the key with maximum normalized weight before the distribution is revealed. The grade tests routing arithmetic, not whether a learned model has discovered a useful feature.

The reference artifact binds Q/K/V meanings and shapes, model and source revisions, ordered positions, causal-mask equality, square-root scaling, stable-softmax convention, numeric resolution, bounds, scope, and owners. Concrete frozen evidence is reconstructed at the public boundary. The material digest covers every weight and context, so the decision is reproducible rather than a detached PASS label.

1def format_example() -> str:
2    report = audit_attention(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_EVIDENCE)
3    return "\n".join(
4        (
5            "example=illustrative_only",
6            f"contract_id={report.contract_content_id}",
7            f"evidence_id={report.evidence_content_id}",
8            f"mask={report.mask_semantics}",
9            f"query0_weights={','.join(f'{value:.3f}' for value in report.weights[0])}",
10            f"query1_weights={','.join(f'{value:.3f}' for value in report.weights[1])}",
11            f"query1_context={','.join(f'{value:.3f}' for value in report.contexts[1])}",
12            f"material_id={report.material_evidence_id}",
13            f"decision={report.decision}",
14        )
15    )

example=illustrative_only
contract_id=attention-contract@sha256:63e4c3c48a6507eb11d35f8a3a47b63b79ecb63ea80eca8d0447733d5a6399d9
evidence_id=attention-evidence@sha256:16a3bf685c8a4d8318cabfcff6be1833bbbb20e0f873f3f8b4e7efdc0d439f55
mask=causal-key-position-lte-query-position-v1
query0_weights=1.000,0.000
query1_weights=0.330,0.670
query1_context=3.302,13.395
material_id=attention-material@sha256:f859dd199d6659edef1507c4e2a13d95dcc8cd71397222fd9869d6b3da282b3b
decision=PASS

Sample routing traces because full attention is quadratic in sequence length and can expose sensitive inputs. Prefer aggregate health signals by layer and request cohort, then capture a bounded redacted replay when an incident gate fires. The trace must include model, tokenizer, sequence, mask, kernel, precision, and evidence identities.