Distributed Training Without Topology Amnesia

Parameters are only one part of the resident set. Training can also hold gradients, optimizer moments, master-precision weights, activations retained for backward, temporary kernel and collective buffers, distributed metadata, and data-loader queues. Some state persists across the run, some exists only within a layer or microbatch, and some can be recomputed. Record bytes, precision, lifetime, ownership, reconstruction operation, and checkpoint responsibility per rank instead of multiplying parameter count by one convenient factor.

Data-parallel ranks process different examples and combine update information. Tensor-parallel ranks cooperate inside operations. Pipeline stages own ordered model regions and exchange activations and gradients. A sharding policy then describes which state is partitioned within a group; it is not another synonym for data parallelism. Write named process groups and rank coordinates explicitly so the same rank is not accidentally assigned contradictory ownership.

1. 01Name each process groupList rank membership, ordering, device and node placement, transport path, expected payload, and the training phase that invokes the group.
2. 02Record collective orderAll ranks must participate consistently; mismatched calls, counts, or datatypes can hang, crash, or corrupt execution rather than returning a friendly validation error.
3. 03Measure the critical pathSeparate overlapped from exposed communication, inspect tails rather than only averages, and correlate each collective with the layer, bucket, microbatch, and slow rank.
4. 04Test contention and topology lossCo-located jobs, storage traffic, bad links, topology fallback, and one slow device can change collective behavior without changing world size.

A bandwidth formula is a hypothesis because collective algorithms, message sizes, topology, protocol, channels, contention, and overlap change the achieved result. Treat NCCL or another communication runtime as part of the versioned training system, and capture its topology and diagnostic output when debugging a hang or regression.

1def audit_training_topology(contract: TrainingContract, evidence: TrainingEvidence) -> TrainingAudit:
2    contract = validate_record(contract, TrainingContract)
3    evidence = validate_record(evidence, TrainingEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another training contract")
6    # Reconstruct rank placement, state shards, and collective order before
7    # applying readiness, memory, throughput, collective, and skew gates.

example=illustrative_only
decision=PASS_TOPOLOGY_AUDIT
topology=dp8-tp2-pp1;world_size=16
state_shards=parameters:8,gradients:8,optimizer:8
peak_memory_gib=58.000;step_throughput=120.000
collective_p95_ms=35.000;rank_step_skew=0.060
claim=LOCAL_TRAINING_AUDIT_NOT_DISTRIBUTED_RUN

The invented fixture maps sixteen ranks evenly onto two eight-device nodes and audits a `dp8-tp2-pp1` full-shard topology. Parameters, gradients, and optimizer state are each sharded eight ways, the declared collective sequence is parameter all-gather followed by gradient reduce-scatter, and checkpoint restore, rank health, telemetry, activation checkpointing, memory, throughput, collective tail, and rank skew all meet local gates. The implementation rejects wrong world size, uneven placement, partial state sharding, reordered collectives, stale scope, tampered digests, exact-type violations, constructor bypass, and caller mutation.

No process group is initialized and no collective, training step, checkpoint, or recovery is executed. Aggregate invented measurements cannot establish convergence, numerical equivalence, reproducibility, or production throughput. The audit demonstrates reviewable topology evidence only.

A checkpoint is a distributed commit, not a directory that happens to contain files. Publish a manifest only after every required shard and metadata record is durable and digest-verified. Test restoration on the same topology and, if supported, an explicitly different topology. Verify optimizer, scheduler, scaler, RNG, data sampler, global step, and model output behavior—not merely that files can be deserialized.

Track step phases and useful samples or tokens processed, not only accelerator utilization. Record data wait, forward, backward, recomputation, optimizer, collective, checkpoint, and idle time per rank. Monitor reserved and peak memory, allocator retries, gradient norms, loss scale, NaNs, stragglers, network retransmits, storage throughput, restart count, recovery point, wasted compute, energy, and cost. The slowest synchronized rank and the least reliable state boundary define the system.

• Keep exact run manifests for code, model, data, tokenizer, optimizer, schedule, precision, kernels, communication runtime, topology, seeds, and environment.
• Alert on rank-level skew and exposed communication before cluster averages hide them.
• Bound checkpoint frequency from recovery-point objective, save duration, storage load, and expected wasted work; more frequent is not automatically safer.
• Requalify after model shape, sequence, batch, precision, optimizer, framework, compiler, driver, communication, cluster, or storage changes.