Assemble and Test a Transformer Block

The residual stream has a declared hidden width d and sequence length T. Attention consumes normalized rows, constructs queries, keys, and values, routes visible values, and returns d features per position. The MLP transforms each position independently and also returns d features. Residual addition is legal only because both branches return the same outer shape and semantics.

1. 01Normalize before the branchFor a pre-norm contract, calculate per-row mean and variance, apply the declared epsilon, gain, and bias, then project Q, K, and V.
2. 02Construct the visible setQuery i may use non-padding keys j where j ≤ i. A future token must not affect any earlier output.
3. 03Apply stable softmaxSubtract the largest visible score before exponentiation and use an accurate sum for the denominator.
4. 04Close the residual pathProject the routed value to d features, validate numeric bounds, and add it to the exact input row.

The position-wise MLP shares parameters across positions but does not mix positions. Attention mixes permitted positions; the MLP transforms features within each position. Residual addition preserves a direct path around each learned branch. Dropout, multiple heads, bias terms, rotary position operations, gated activations, and fused kernels are deliberately outside this minimal executable surface and must be named when added.

1def audit_transformer_block(
2    contract: TransformerBlockContract,
3    weights: TransformerBlockWeights,
4    evidence: TransformerBlockInput,
5) -> TransformerBlockReport:
6    """Validate scope and run one bounded pre-norm causal block."""

example=illustrative_only
contract_version=transformer-block-v1
contract_content_id=transformer-contract@sha256:1e518b5f80e3143f2e5ec1ff73c93c007852c676206ed833cda980c23eddbeec
weights_content_id=transformer-weights@sha256:3741140d92a7d319759d7ce417b486c01ea20c8cd4d9bd2cd5355b999e0896d5
input_content_id=transformer-input@sha256:4f7c85be333eba606a98d3bb4bcdd24ccaef020d0b9c077256ba9184919fcac4
shape=3x2
mask=causal-plus-padding
norm=pre-layer-norm
status=PASS
claim=TEACHING_BLOCK_NOT_PRODUCTION_KERNEL

The excerpt is literal artifact source. The public boundary requires concrete frozen dataclasses, revalidates input and parameter tuples, checks model and contract scope, recomputes weight and evidence digests, caps dimensions and parameter count, rejects booleans and non-finite or sub-resolution values, and bounds every derived output. Its stable softmax subtracts the maximum visible score and its sums use math.fsum.

1. 01Perturb the futureChange a later token sharply and require earlier outputs to remain unchanged under causal masking.
2. 02Forge the scopeReuse values under a stale model version, architecture digest, weight content ID, or input digest and require refusal.
3. 03Attack paddingReject a non-contiguous padding suffix and verify the declared padded-query output behavior instead of leaving it implicit.
4. 04Attack arithmeticInject bool, NaN, infinity, subnormal inputs, excessive magnitudes, unsafe epsilon, and derived overflow.
5. 05Attack representationPass mutable lists, duck types, ragged rows, and constructor-bypassed records at the public boundary.

Release a block change with old/new checkpoint replays, fixed token batches, attention-mask probes, forward tolerances by precision, local VJP or gradient checks, training smoke tests, and end-to-end task evaluation. Keep compiler, kernel, device, precision, seed, and framework versions with the evidence. A fast fused path should have a readable reference oracle even when production never runs the oracle.

A passing report supports this narrow statement: one concrete pre-norm, single-head, causal teaching block accepted its declared immutable inputs and weights, preserved the expected shape, and completed bounded binary64 arithmetic. It does not prove mathematical equivalence to every transformer, production-kernel parity, correct gradients, training convergence, or model quality.