Neural Training Incident: Loss Becomes NaN

The laboratory is a general contract-ordering exercise: it grades only the first failed stage across ingress, batch, forward, backward, optimizer, and validation. It does not ask for a containment choice and it is not the artifact's numeric telemetry schema. The executable runbook later narrows the path to input, forward, loss, backward, and optimizer, where non-finite counts determine a HOLD action.

1. 01Hold the updateDo not step the optimizer or silently skip forward without recording the decision. A non-finite gradient can contaminate moments and parameters.
2. 02Preserve the boundaryCapture run and step IDs, checkpoint, optimizer, scheduler, scaler, batch identity, loader cursor, random state, environment, and distributed topology.
3. 03Mark the last known-good stateRetain a recovery point whose model, optimizer, scheduler, scaler, and data position were verified together.
4. 04Choose replay authorityAssign one incident owner and stage owners; avoid multiple responders changing learning rate, data, and precision at once.

Check complete tensors or a declared, bounded summary whose finite and non-finite element counts reconcile. A finite maximum alone cannot prove a tensor is finite because a reduction may ignore NaN, sample the wrong slice, or run after the value was replaced. Name the tensor set and probe source at every boundary.

1. 01Autocast selected operationsReduced precision can accelerate safe kernels while reductions or sensitive functions remain at a wider type.
2. 02Scale before backwardA declared scaler multiplies the loss so small gradient components are less likely to underflow.
3. 03Unscale before clipping or inspectionThresholds and finite checks intended for real gradients must see the unscaled magnitude unless explicitly defined otherwise.
4. 04Step only after qualificationIf the framework detects non-finite gradients, record the skipped update and scaler transition as evidence rather than calling the run recovered.

1    actions = {
2        "input": "hold-and-inspect-input-data-contract",
3        "forward": "hold-and-replay-forward-activations",
4        "loss": "hold-and-inspect-objective-reduction",
5        "backward": "hold-and-inspect-gradient-path-and-loss-scaling",
6        "optimizer": "hold-and-inspect-optimizer-state-and-update",
7    }
8    return IncidentReport(
9        contract_content_id=contract.content_id,
10        evidence_id=evidence.evidence_id,
11        evidence_content_id=evidence.evidence_content_id,
12        decision="HOLD" if earliest is not None else "CONTINUE_BOUNDED_DIAGNOSTICS",
13        earliest_nonfinite_stage=earliest,
14        action=actions[earliest] if earliest is not None else "no-nonfinite-evidenced-check-other-signals",
15        root_cause_claim="NONE_SYMPTOM_LOCATION_IS_NOT_CAUSALITY",
16    )

example=illustrative_only
contract_version=training-incident-v1
run=run-2026-08-25-001
step=1842
evidence_content_id=training-incident-evidence@sha256:9e972d95a21289528822637d157b32eee66be582d614bc2ed5d4beb6e08ffcc9
earliest_nonfinite=backward
decision=HOLD
action=hold-and-inspect-gradient-path-and-loss-scaling
root_cause_claim=NONE_SYMPTOM_LOCATION_IS_NOT_CAUSALITY

The excerpt is literal source. The artifact requires concrete immutable records for all five stages, validates their exact order, owners, reconciled element counts, and observed/not-run transitions, and binds the run, step, model, data, batch, objective, optimizer, precision, loss-scale evidence, and probe sources. It rejects duplicate telemetry and tensor-set identities, recomputes the complete evidence hash, revalidates constructor-bypass mutations, caps per-stage and total work, and rejects booleans, NaN, infinity, sub-resolution summaries, unsafe scales, and excessive finite magnitudes. A later stage can be explicitly NOT_RUN only after an earlier non-finite boundary.

1. 01Reproduce without an updateLoad the complete pre-step state and failing batch, enable deterministic settings where feasible, and confirm the first bad boundary.
2. 02Shrink the graphRun one example, then one layer or loss term around the earliest boundary while preserving the triggering values.
3. 03Change one mechanismCompare precision, loss scale, reduction, input record, or implementation one at a time under the same replay contract.
4. 04Inspect upstream finite extremesA stage can remain finite yet enter a range that guarantees the next exponential, division, square, or reduction will overflow.
5. 05Prove recoveryResume from a known-good state, replay the fixed case, run a bounded continuation, and verify quality and state integrity before reopening the run.

• Add a named finite/range assertion immediately before and after the confirmed failing operation, with bounded sampling overhead.
• Create a regression fixture from sanitized triggering values and test the exact precision, reduction, scaler, and optimizer order.
• Checkpoint model, optimizer, scheduler, scaler, random, and data-loader state atomically enough for the promised recovery level.
• Alert on finite extremes and skipped updates before aggregate loss becomes non-finite, while retaining stage and slice support.
• Document rollback, quarantine, resume, and data-remediation authority so incident containment does not become an improvised product decision.

The runbook performs O(S) validation for five stage summaries and stores O(S) report evidence; collecting those summaries can be O(number of inspected elements). The contract therefore caps each stage and the total. Real incident tooling should balance complete finite checks at critical boundaries with sampled distributions elsewhere, then retain full tensors only under controlled diagnostic capture.