LoRA and the Low-Rank Update Hypothesis

The factorization constrains which matrix changes training can express. Multiplying the two factors gives a full-shaped update, but its columns and rows are coupled through r latent directions. This can be effective when the necessary adaptation lies in a small subspace. It is not a theorem that every domain shift, behavior change, or modality transition fits that subspace. The useful hypothesis is narrower: this base, these targets, this rank, and this objective may be sufficient for the measured task.

A common initialization makes one factor random and the other zero so the initial update vanishes, preserving the base function at the start. Training changes the factors, not the frozen base. At deployment a compatible unquantized update may be merged into a base copy, or the factors may remain separate for adapter switching. Merging, quantization, and routing change operational behavior and require their own parity tests; this budget does not perform them.

For an illustrative 128×128 matrix and rank 8, A has 1,024 parameters and B has 1,024: 2,048 trainable parameters rather than 16,384 full-matrix entries. Applying this to two named projections gives 4,096 parameters. Biases, embeddings, output heads, or other fully trainable modules must be counted separately if enabled; the fixture excludes them. A phrase such as 'attention targets' is not precise enough because architectures differ in projection shapes and sharing.

Resolve target paths against the actual base checkpoint. Enforce unique IDs, exact dimensions, ordered manifests, and rank bounds. A stale path that selects no module can appear wonderfully cheap while adapting nothing. A duplicated target can double-count state or misconfigure training. The contract checks named target identity and total target entries against the declared base size; an integration must additionally verify those identities against a real model manifest.

Integer byte accounting avoids rounding a small adapter into an apparently zero cost. The artifact returns a global lower bound, not a per-device budget: sharding divisibility is checked, but placement, replication, optimizer partitioning, and communication buffers are not modeled. Dividing the final number by the GPU count would invent an unsupported distribution policy. Peak memory must be measured in the actual runtime with the intended sequence lengths, batch sizes, optimizer, and checkpointing configuration.

Increasing rank increases the count linearly for fixed target shapes, but quality need not increase monotonically. Adding a target changes which transformations can adapt, not merely how many parameters exist. Predict the count before revealing the budget, then ask whether the bound evaluation establishes sufficiency. A configuration can be comfortably below a memory limit and still fail the task.

The fixture uses a one-million-parameter synthetic base, two 128×128 projections, rank 8, BF16 weights, and the declared FP32-state AdamW policy. Its 4,096 adapter parameters require 65,536 bytes of trainable state; adding two million base-weight bytes gives a 2,065,536-byte lower bound. Its invented quality score of 0.8 misses the invented requirement of 0.9. The report blocks sufficiency despite the small budget and explicitly excludes activations, workspaces, and runtime memory.

1def main():
2    result = audit(example_contract())
3    print("example=illustrative_only")
4    print(f"trainable_parameters={result.trainable_parameters}")
5    print(f"lower_bound_bytes={result.lower_bound_bytes}")
6    print("decision=" + result.decision)
7    print("claim=" + result.claim)

example=illustrative_only
trainable_parameters=4096
lower_bound_bytes=2065536
decision=BLOCK_LOW_RANK_INSUFFICIENT
claim=LOWER_BOUND_EXCLUDES_ACTIVATIONS_WORKSPACES_RUNTIME

Frozen target records and copied identity tuples are revalidated at audit entry. A configuration digest binds rank, target shapes and order, adapter IDs, precision, optimizer, base count, version, and sharding assumptions. Evaluation also binds model, data, scope, case count, and external result content identity. Changing rank without new evaluation evidence is therefore not a valid reuse of the old sufficiency claim. The artifact validates these attestations; it does not execute a model or authenticate the referenced run.

A failed quality gate does not identify rank as the cause. First rule out data defects, wrong target paths, incompatible templates, optimization instability, and an inadequate base model. Then compare controlled rank and target sweeps under the same data and evaluation protocol. If increasing rank or broadening target coverage repairs a stable failure, you have evidence about the constraint. If every rank fails similarly, buying more low-rank parameters may not address the actual deficit.

Compare against a stronger base and, where justified and feasible, a full-fine-tuning control. Watch for forgetting outside the tuned domain and failures on underrepresented cohorts. Stop treating low rank as sufficient when qualified evidence says it misses requirements; choose a new experiment or intervention through the adaptation decision process. Do not relabel a failed requirement as acceptable merely because the adapter is convenient to deploy.