Multi-Head Attention Is Parallel Representation Routing

Each head receives the same sequence of model-width representations but owns distinct Q, K, and V projection parameters. The projections select and recombine input coordinates into a head-specific compatibility space and payload space. The head then performs the same scaled routing operation derived in the previous lesson.

A common configuration chooses dk = dv = dmodel/h, keeping the concatenated width equal to dmodel. That is a design convention, not a mathematical requirement. Grouped-query and multi-query attention further change how query heads share K/V projections. Always derive the actual checkpoint shapes instead of inferring them from a generic Transformer diagram.

Heads normalize their own score rows, so one head cannot directly trade probability mass with another. Their payloads meet only after independent routing, through concatenation and WO. Consequently, a head's apparent pattern can be amplified, cancelled, or remixed by the output projection and residual stream.

1. 01Project per headBind matrix contents and projection versions; matching shapes cannot detect stale or swapped parameters.
2. 02Route independentlyApply the same bound mask, scale, and stable softmax to each head while retaining head identity.
3. 03Concatenate in declared orderPreserve the mapping from head output coordinates to output-projection rows.
4. 04Project and rejoin the residual pathInterpret head evidence in the context of WO and downstream computation, not in isolation.

A strong specialization study defines a task metric and reference population before inspecting heads, evaluates held-out examples, intervenes on one or matched sets of heads, includes random and magnitude-matched controls, measures confidence intervals, and repeats across checkpoints or seeds. Even then, name the bounded behavior rather than assigning an anthropomorphic job title.

The reference artifact validates concrete frozen head records, unique head IDs, Q/K/V and output-projection shapes, model/layer revisions, mask and scale semantics, bounded input representations, operation count, provenance, and scope. It emits separate content identities for the contract, projection collection, input evidence, each head output, and material combined result. Its policy always reports that head existence does not establish specialization.

1def format_example() -> str:
2    report = audit_multi_head_routing(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_EVIDENCE)
3    return "\n".join(
4        (
5            "example=illustrative_only",
6            f"contract_id={report.contract_content_id}",
7            f"projection_id={report.projection_content_id}",
8            f"head_ids={','.join(head.head_id for head in report.heads)}",
9            f"token0_output={','.join(f'{value:.3f}' for value in report.combined_output[0])}",
10            f"specialization={report.specialization_conclusion}",
11            f"required_evidence={report.required_specialization_evidence}",
12            f"material_id={report.material_evidence_id}",
13            f"decision={report.decision}",
14        )
15    )

example=illustrative_only
contract_id=multi-head-contract@sha256:a11697bdba26208873482329d54a60fd0866d4c0129aaedb2a27d8be91575534
projection_id=multi-head-projections@sha256:208a126e380bfd2a5c0546f6bfa6900bd8f77303d8a4bc20a5b7bc6fe7afacc5
head_ids=head-0,head-1
token0_output=0.731,0.500
specialization=NOT_ESTABLISHED_FROM_HEAD_EXISTENCE
required_evidence=TASK_METRICS_PLUS_CONTROLLED_HEAD_ABLATION
material_id=multi-head-material@sha256:9578fa762103f264813d58b008406ae7fafdfb954425f58d9880b22d76c1ad97
decision=PASS

Do not log every full attention map by default. Aggregate entropy, concentration, and inter-head similarity on sampled, access-controlled traces; join anomalies to task outcomes and model identities. A low-diversity alert is a prompt for investigation, not an automatic pruning command.