Matrices as Small Programs

Row i is a small program for output yᵢ: multiply each input by the corresponding row weight, then add. Column j exposes every path by which input xⱼ can affect the outputs. Zeros remove direct influence; repeated or correlated rows can make outputs redundant; fewer rows than columns can discard distinctions.

A transpose reverses rows and columns; it is not a visual formatting operation. In a learned layer, transposing can change whether examples, tokens, channels, or features are being mixed. Write axis names beside shapes during derivation, then make the executable boundary enforce them.

The reference artifact represents every stage with immutable weights plus unique input and output axis names. A pipeline accepts adjacent stages only when the complete semantic axis tuple matches. It records intermediate values for diagnosis and compiles the stages into one matrix, allowing a test to prove that the optimized and inspectable executions agree.

1if __name__ == "__main__":
2    run = EXAMPLE_PIPELINE.run((3.0, 1.0))
3    compiled = EXAMPLE_PIPELINE.compile()
4    print(f"pipeline={EXAMPLE_PIPELINE.name}")
5    print(f"compiled_shape={compiled.shape[0]}x{compiled.shape[1]}")
6    print("stage_outputs=" + ";".join(
7        f"{trace.stage}:{trace.values}" for trace in run.stages
8    ))
9    print(f"compiled_matches={compiled.apply(run.input_values) == run.output}")

pipeline=rotate-scale-project
compiled_shape=1x2
stage_outputs=rotate:(-1.0, 3.0);scale:(-2.0, 1.5);project:(1.0,)
compiled_matches=True

1. 01Declare axesName dimensions by meaning rather than relying on position: raw features, hidden channels, heads, tokens, or decision outputs.
2. 02Run with stage tracesKeep an inspectable path that reveals which transformation first produced a bad value or erased a distinction.
3. 03Compile and compareCompose compatible linear stages for efficiency, then test the compiled map against the staged path over representative and adversarial inputs.

Learned weight matrices project input features into hidden spaces, mix channels, produce attention queries, keys, and values, and map hidden states back to outputs. The same rules apply, but tensors add batch, sequence, head, and channel axes. A reshape or transpose changes which collection a matrix operates over, so annotate the contraction axes rather than saying only that two tensors are multiplied.

1. 01Swap non-commuting stagesUse anisotropic scale and rotation so a reorder visibly changes the result. Identity-like examples can let the bug escape.
2. 02Connect equal shapes with different meaningsVerify that named axes fail closed even when raw dimensions match exactly.
3. 03Transpose one boundaryCheck whether examples, tokens, or channels are now mixed. Assert the entire expected shape and axis signature after each operation.
4. 04Inject ragged and non-finite weightsReject partial rows, booleans, NaN, and infinity at capture time so they cannot contaminate later stages.
5. 05Mutate the caller's buffersConfirm that compiled behavior cannot be rewritten through a list retained by configuration code.

• Emit resolved tensor shapes, axis conventions, dtype, and weight revision into diagnostic metadata without logging sensitive values by default.
• Test fused, quantized, exported, and accelerator-specific paths against a trusted implementation with dtype-aware tolerances.
• Monitor saturation, non-finite rates, norm changes, and output drift by model version and slice.
• Keep a staged diagnostic path or sampled intermediate hooks even when production compiles operations into one kernel.
• Benchmark arithmetic and memory movement separately; algebraic fusion can reduce traffic while changing rounding behavior.