Optimization Under Noise
A jagged loss curve is not a diagnosis. Read stochastic optimization through direction, variance, step scale, and fixed-source objective evidence before changing the optimizer.
- Authorship
- InterviewsVector
- Published / updated
- 2026-08-14 / 2026-08-14
- Review status
- Artifact tests passing · primary sources recorded
Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.
The decision in one pass
A mini-batch gradient is an estimate of a direction, so individual steps and batches may move uphill while a training window still makes useful progress. Diagnose the window, not one point: bind the run to an objective ID, dataset revision, sampler ID, parameter version, batch size, seed, precision, learning rate, owners, and probe source; compare the mean sampled gradient with an independent fixed-source reference; measure variance relative to signal; bound the parameter-relative step; and require both net progress and a versioned objective-trace consistency envelope. Non-finite values, changed identities, wrong shapes, missing owners, weak improvement frequency, excessive upward excursion, overflow, or underflow invalidate the evidence. High variance with sound direction, scale, provenance, and consistent progress calls for noise or rate work, not an automatic optimizer rewrite.
Why this matters
Stochastic training telemetry is easy to overread. Teams stop healthy runs because one batch rises, continue broken runs because a smoothed chart drifts down, or change batch size and learning rate together until the causal signal disappears. At scale, that burns compute and can publish a model whose apparent recovery came from a changed data stream rather than a repaired update rule.
You will be able to
- Derive the stochastic-gradient update and state the assumptions hidden inside an unbiased-gradient claim.
- Separate gradient direction, noise magnitude, step scale, and observed objective progress into distinct diagnostic signals.
- Use windows and a fixed, identity-bound probe source without smoothing away instability.
- Reject shape, precision, sample-count, identity, ownership, and finite-number violations before computing a diagnosis.
- Choose an operational response that preserves causal evidence instead of changing several controls at once.
Prerequisite contract
- Matrices as Small Programs →
- Probability for Decisions, Not Decoration →
- — Derivatives, norms, means, and variance
Your Vector Loop for this lab
- 01
Model
Treat each mini-batch gradient as a versioned observation of a latent optimization signal.
- 02
Derive
Relate mean direction, variance, alignment, learning rate, parameter scale, and objective movement.
- 03
Build
Implement a fixed-window diagnostic whose configuration and evidence share one content identity.
- 04
Stress
Reverse directions, drift batch sizes, inject non-finite values, flatten progress, and overflow the step.
- 05
Operate
Run controlled probes, retain raw evidence, and change one optimization control at a time.
- 06
Defend
Explain why a noisy trace can be healthy and why a smooth trace can still be invalid evidence.
Model a stochastic gradient as an evidence contract
For parameters θ and per-example loss ℓ(θ; z), the training objective is an expectation over a named data distribution. A mini-batch replaces that expectation with a sample average. Calling the result stochastic does not make every source of variation legitimate: stale parameters, duplicated examples, a changed augmentation policy, mixed data revisions, or a reduction bug are contract failures, not optimization noise.
θ(t+1) = θ(t) − η(t) g(t), with E[g(t) | θ(t)] ≈ ∇L(θ(t))
The conditional expectation is an assumption to test against the sampling and systems path. The learning rate η scales both useful signal and noise; it cannot repair a direction whose meaning has drifted.
| Variation source | Legitimate stochasticity? | Evidence to inspect |
|---|---|---|
| different examples from one declared sampler | possibly | sample identities, inclusion policy, batch size, seed |
| data revision changes mid-window | no | dataset and sampler identity on every observation |
| asynchronous gradient computed on stale parameters | separate systems effect | parameter version and staleness distribution |
| dropout or augmentation draw | possibly | randomness stream and transform version |
| NaN replaced with zero | no | finite-value rejection before aggregation |
Read raw steps and bounded windows together
Noisy progress or broken learning?
Adjust deterministic learning-rate, noise, and step controls; compare raw log loss with a four-step window; inspect opening and ending evidence plus uphill-step count; then classify noisy progress versus broken or stalled behavior and complete a graded diagnosis check.
Diagnose the trend beneath gradient noise
This deterministic run optimizes a one-dimensional quadratic. Judge learning from a windowed loss trend—not from whether every update is downhill.
- Opening loss window
- 16.7
- Ending loss window
- 0.133
- Uphill steps
- 13
The same fixed pseudo-noise sequence is used on every run. Changing a control changes the assumptions, never the sample luck.
Noisy progress
Several updates move uphill, yet the ending window is materially lower. Noise changes the path without erasing the learning signal.
Make a prediction, then check it against the current evidence.
Raw points expose spikes and non-finite transitions; a bounded window exposes direction over more than one draw. Keep both. A moving average without its raw series can hide alternating divergence, while a single uphill step can be entirely consistent with a descending expectation. The window length is part of the measurement contract, not a visual preference selected after seeing the curve.
Derive complementary signals before assigning a cause
ḡ = (1/W) Σᵢ gᵢ; S = [(1/W) Σᵢ ||gᵢ − ḡ||²] / max(||ḡ||², ε²)
The artifact reports S as a simple dimensionless noise proxy across W sampled gradients. It is useful for comparison under a fixed contract, but it is not a universal convergence theorem or a direct critical-batch-size prescription.
alignment = (ḡ · gref) / (||ḡ|| ||gref||)
A reference gradient from a fixed, separately identified probe source tests direction. Reusing one sampled mini-batch as both observation and reference would make the check circular.
relative step = ||η ḡ|| / max(||θ||, δ)
Parameter-relative scale catches a finite but destructive update. The floor δ and the threshold are versioned policy choices, especially important near zero-valued parameters.
consistency = improving steps / W; excursion = max(0, L(t+1) − L(t)) / max(|L(0)|, κ)
Positive end-to-end progress is insufficient if most steps worsen or the path contains a catastrophic spike. The minimum improving fraction, maximum upward-excursion ratio, and scale floor κ are identity-bound policy thresholds.
| Observed pattern | Plausible interpretation | Next controlled check |
|---|---|---|
| good alignment, positive window progress, high S | signal exists but sampling variance is large | hold data fixed; compare batch or rate one at a time |
| bad alignment with a trusted probe | sign, target, sampling, staleness, or gradient bug | stop and reproduce the exact window |
| good alignment, oversized relative step, worsening objective | direction may be sound but update scale is unsafe | reduce only the rate and replay |
| smooth curve after source identity changed | comparison is invalid | split the run at the identity boundary |
| near-zero reference norm | probe has insufficient directional signal | choose a declared fallback test; do not invent a cosine |
Build a diagnostic that fails before it guesses
1 broken_signal = (2 alignment < contract.min_alignment3 or objective_progress <= 0.04 or relative_step > contract.max_relative_step5 or improving_fraction < contract.min_improving_fraction6 or max_objective_excursion_ratio > contract.max_objective_excursion_ratio7 )8 if broken_signal:9 decision = "STOP_AND_INVESTIGATE"10 elif noise_scale > contract.max_noise_scale:11 decision = "REDUCE_NOISE_OR_RATE"12 else:13 decision = "CONTINUE"Expected output
example=illustrative_only
contract_version=noisy-optimizer-v1
window=window-0001
gradient_mean=(1.000000,-0.500000)
alignment=1.000
noise_scale=0.025
relative_step=0.079
objective_progress=0.240000
improving_steps=3/4
max_objective_excursion_ratio=0.015
decision=CONTINUEVerify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/noisy_optimization.
The code block is a literal contiguous excerpt from the downloadable artifact, including its function indentation. Before reaching it, the implementation requires concrete frozen OptimizationContract and OptimizerWindow records, rejecting mutable duck-typed aliases, then verifies the contract digest, stable parameter names, binary64 execution, exact vector widths, exact batch counts, objective ID, dataset revision, sampler ID, parameter version, fixed probe identity, finite parameters, gradients, and objective boundaries, accountable owners, and non-overflowing, non-underflowing step arithmetic. The improving-fraction and excursion thresholds are versioned policy, and the displayed values are invented fixtures rather than recommendations or benchmark evidence.
Attack the evidence path, not only the learning rate
- 01Reverse the reference directionProve that positive-looking objective movement cannot override a clear direction mismatch without an explicit investigation.
- 02Drift one batch sizeReject the entire window. A variance comparison with a different sampling scale is not the contracted measurement.
- 03Inject NaN and infinityFail before norms or means turn invalid values into plausible status text.
- 04Overflow and underflow the updateExercise both a huge finite learning rate and the smallest representable positive rate; neither may silently become an unusable step.
- 05End above the opening objectiveEven several improving substeps cannot convert negative window progress into a continue decision.
- 06Hide a catastrophic path behind a slightly better endpointUse objective values (2, 1000, 10000, 100000, 1.9). The positive endpoint delta cannot pass the minimum improving-fraction or maximum excursion gates.
- 07Change only an owner or source IDConfirm that operational accountability and evidence provenance change the immutable contract identity.
Operate optimization as controlled change
| Retain per window | Why | Response when it changes |
|---|---|---|
| run, code, data, sampler, probe, and contract IDs | preserve comparability | split the series and requalify |
| raw loss plus declared window summary | show spikes and trend | inspect both before tuning |
| gradient norm, alignment, and noise proxy | separate direction from variance | choose a targeted experiment |
| parameter norm and relative step | surface scale instability | hold other controls fixed |
| throughput, skipped updates, and finite-value failures | connect math to runtime | stop or quarantine invalid workers |
Use a short replayable window before committing a long run. When a gate fails, preserve the checkpoint, inputs, and raw observations. Change one factor—rate, batch size, clipping rule, sampler, or precision—then replay the same probe. Scaling to more workers adds staleness, reduction order, dropped work, and data-overlap questions; those need their own evidence rather than being labeled more noise.
Operate at three altitudes
Production lens
- — Log contract, sampler, data, code, checkpoint, probe, precision, and parameter-version identities with every diagnostic window.
- — Preserve raw observations beside smoothing; alert on non-finite values, skipped updates, alignment failure, relative-step breach, and source drift.
- — Replay a bounded fixed-source probe after changes to learning rate, batch size, clipping, precision, worker count, or reduction topology.
- — Keep optimization qualification separate from validation and product metrics; a healthy training objective does not prove useful generalization.
Staff lens
- — Define who owns the objective, gradient implementation, sampler, data revision, distributed reducer, and stop authority.
- — Budget experiments so a diagnosis can change one factor at a time instead of forcing coupled emergency tuning.
- — Treat worker staleness, duplicate samples, and reduction nondeterminism as system contracts with measurable envelopes.
- — Require checkpoint lineage to record failed and overridden optimization gates, not only the final successful run.
Interview defense
Training loss is jagged and occasionally rises. How do you decide whether SGD is working?
I would first verify that the run is comparable: same loss, data and sampler identity, batch size, seed policy, precision, checkpoint, and reduction semantics, with finite values and no skipped or stale updates. Over a predeclared window I would retain raw loss, compare the mean sampled gradient with a fixed-source reference, estimate variance relative to signal, bound the update relative to parameter scale, and check opening-to-ending objective movement. A few uphill batches can be healthy. Bad direction, an oversized step, no window progress, or invalid identity means stop and reproduce. High variance with sound direction and progress motivates a controlled batch or learning-rate experiment, one change at a time.
Expect the interviewer to press on
- — Why is a smoothed loss curve insufficient?
- — What changes when gradients are asynchronous across workers?
- — How would you distinguish a sampling bug from an aggressive learning rate?
Misconceptions to remove
“Any uphill mini-batch means gradient descent is broken.”
A stochastic update can increase one observed batch loss while improving the declared objective across a bounded window. Inspect direction, scale, and comparable window evidence together.
“A smooth decreasing training curve proves the optimizer is correct.”
Smoothing can hide alternating instability, and a consistently biased gradient can decrease the wrong or leaked objective. Preserve raw data and independent validation.
“The gradient noise scale directly tells us the optimal batch size.”
Noise statistics depend on estimator, objective, point in training, and theoretical assumptions. The artifact's simple proxy is a comparative diagnostic under one fixed contract.
Check your model
1. Why must the reference gradient use a separately identified fixed source?
Otherwise the same random batch can define both the observation and the expected direction, creating a circular alignment check that cannot expose sampling or sign failure.
2. A window has good alignment and positive progress but exceeds the noise threshold. What is the artifact's response?
REDUCE_NOISE_OR_RATE. The signal is not labeled broken; the next step is a controlled variance or rate experiment under the same evidence contract.
3. Why include parameter norm in a step diagnostic?
A finite update can still be enormous relative to the current parameter scale. The relative step reveals that mismatch and makes the policy portable across differently scaled parameters.
Prove the mechanism
Instrument a small stochastic optimization loop with a frozen probe source. Produce immutable window records with exact shapes, batch counts, data and sampler identities, precision, owners, raw objective boundaries, sampled gradients, and parameter values. Add decision tests for good noisy progress, wrong direction, excess variance, no progress, non-finite input, stale identity, and update overflow or underflow.
Add a production constraint
Extend the diagnostic to distributed workers. Bind parameter version and sample identities to each contribution, measure staleness and overlap, compare deterministic and alternate reduction orders, and design a replay that isolates sampling noise from synchronization failure.
Artifact: Noisy optimizer diagnostic
courses/ai-engineering/reference-impl/noisy_optimization/noisy_optimizer_diagnostic.py
Download reference implementationPrimary references and next links
References
- 1. Optimization Methods for Large-Scale Machine Learning
Bottou, Curtis, and Nocedal. Primary SIAM publication supporting the stochastic-gradient model, step-size reasoning, practical behavior, and noise-reduction framing used in this lesson.
- 2. An Empirical Model of Large-Batch Training
McCandlish et al.. Primary paper motivating joint measurement of gradient signal and variance; the lesson explicitly distinguishes its simple diagnostic proxy from the paper's batch-size model.
- 3. Python math.fsum
Python documentation. Official runtime documentation for the accurate summation primitive used by the reference diagnostic.
Continue through the graph
- Matrices as Small Programs →
Review the linear operations that produce and transform gradients.
- Numerical Stability Is Part of the Algorithm →
Continue from stochastic variation to finite-precision failure.
- Define the Measurement Before the Model →
Connect optimization evidence to predeclared release measures.
Glossary: stochastic gradient · mini-batch · gradient noise · alignment · relative step · probe set · staleness · optimization window