InterviewsVector
Arc 4
Build labIntermediate120 min estimateOriginal publication

Backpropagation as Local Contracts

A computation graph learns because every operation can answer one local question: given pressure on my output, what pressure should each input receive?

Authorship
InterviewsVector
Published / updated
2026-08-11 / 2026-08-11
Review status
Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Backpropagation is reverse-mode automatic differentiation over a computation graph. Each operation stores enough forward context to compute a vector-Jacobian product during the backward pass. Gradients from every downstream path accumulate at shared values; an optimizer uses the resulting parameter gradients only after the graph has finished propagating them.

Why this matters

When a training run stalls or explodes, the useful unit of debugging is not the whole neural network. It is the local contract between values, operations, shapes, derivative rules, accumulation, and precision.

You will be able to

  • Separate backpropagation, gradient descent, and parameter updates.
  • Derive reverse-mode differentiation on a branched graph.
  • Implement scalar autodiff with topological ordering and gradient accumulation.
  • Validate analytical gradients with central finite differences.
  • Diagnose detach, overwrite, saturation, and numerical precision failures.

Prerequisite contract

Your Vector Loop for this lab

  1. 01

    Model

    View learning as credit assignment through a graph of operations.

  2. 02

    Derive

    Compose local derivatives with the chain rule in reverse.

  3. 03

    Build

    Implement a tiny scalar autodiff engine and gradient checker.

  4. 04

    Stress

    Break accumulation, graph connectivity, activations, and precision.

  5. 05

    Operate

    Instrument gradient flow and separate training from optimizer state.

  6. 06

    Defend

    Explain backpropagation without saying the network learns backwards.

The graph owns dependencies; operations own derivatives

The forward pass creates values and records how each result depends on earlier values. The backward pass starts with dL/dL = 1 and visits operations in reverse topological order. An operation never needs to understand the whole model. It receives the gradient of the loss with respect to its output and returns contributions for its inputs.

Local derivative contracts

A branched graph shows forward values moving right and gradient contributions accumulating left at a shared parameter.

Forward values and reverse gradient contracts in a computation graphInputs x and w enter a multiply operation, then bias b enters addition to produce y and a loss. Cyan arrows show the forward pass and amber arrows show local gradients flowing in reverse.FORWARD: compute and retain local contextREVERSE: receive upstream pressure, return local VJPxinputwparameter×save x, wbparameter+sum pathsLloss1∂L/∂m× w, × x
Each operation owns a small derivative contract. Shared downstream paths add their contributions; the optimizer is not part of this graph traversal.

Derive reverse-mode on a branch

y = x² + 3x; dy/dx = 2x + 3

The graph has two paths from x to y. Reverse-mode must add both contributions at x; replacing one gradient with the other is a silent bug.

  1. 01SeedSet the output gradient to one because dy/dy = 1.
  2. 02Reverse x²Contribute upstream × 2x to x.
  3. 03Reverse 3xContribute upstream × 3 to x.
  4. 04AccumulateAdd both contributions: 2x + 3.

vjp(g, x) = gᵀ Jₓ

Tensor frameworks usually propagate vector-Jacobian products rather than materializing full Jacobian matrices. This is why reverse-mode is efficient when a scalar loss depends on many parameters.

Build the smallest engine that can be wrong

local_autodiff.py
1class Value:
2    def __init__(self, data, parents=(), backward=lambda: None):
3        self.data = float(data)
4        self.grad = 0.0
5        self.parents = tuple(parents)
6        self._backward = backward
7 
8    def __mul__(self, other):
9        other = other if isinstance(other, Value) else Value(other)
10        out = Value(self.data * other.data, (self, other))
11        def backward():
12            self.grad += other.data * out.grad
13            other.grad += self.data * out.grad
14        out._backward = backward
15        return out
16 
17    def backward(self):
18        order, seen = [], set()
19        def visit(node):
20            if id(node) in seen: return
21            seen.add(id(node))
22            for parent in node.parents: visit(parent)
23            order.append(node)
24        visit(self)
25        self.grad = 1.0
26        for node in reversed(order): node._backward()
27 
28x = Value(3)
29y = x * x
30y.backward()
31print(y.data, x.grad)

Expected output

9.0 6.0

Verify: Run python -m unittest discover courses/ai-engineering/reference-impl/backprop.

The plus-equals operator is the important character in this implementation. A parameter may influence the loss through thousands of paths. Gradient accumulation is graph semantics, not an optimization detail.

Use finite differences as an independent witness

f′(x) ≈ [f(x + ε) − f(x − ε)] / (2ε)

Central differences provide an implementation-independent estimate. Too-large epsilon adds truncation error; too-small epsilon loses the difference to floating-point cancellation.

SymptomLikely contract failureFirst check
one branch has no effectgradient overwritten, not accumulatedshared-node unit test
all upstream grads are zerodetach or saturated derivativegraph connectivity and activation inputs
gradients double each stepgradients not clearedzeroing boundary
finite difference disagreeswrong local derivative or mutationsmall isolated expression
training diverges only in low precisionoverflow or underflowloss scaling and gradient stats

Break backpropagation deliberately

  1. 01Replace += with =A branched graph returns the last contribution only. A linear chain still passes, which is why branch tests matter.
  2. 02Mutate a saved forward valueThe backward rule differentiates a different computation from the one that produced the loss.
  3. 03Detach an intermediateValues remain numerically plausible while upstream parameters receive no learning signal.
  4. 04Use a saturated activationThe graph is correct, yet local derivatives approach zero and credit cannot travel.

Observe gradient flow in a real training system

  • Track gradient norm distributions by layer, not only a global norm.
  • Record activation saturation, non-finite counts, optimizer step size, and loss scale.
  • Check whether the first bad value appears in data, forward activations, loss, gradients, or optimizer state.
  • Keep a deterministic tiny-batch overfit test; a model that cannot fit a tiny sample has an implementation or optimization problem.
  • Checkpoint model, optimizer, scheduler, scaler, data position, and random state together when exact recovery matters.

Reverse-mode stores forward context for the backward pass, so activation memory grows with graph size. Activation checkpointing trades extra forward compute for lower stored activation memory. It changes the execution plan, not the derivative being computed.

Operate at three altitudes

Production lens

  • — Use non-finite guards before optimizer updates, and preserve the failing batch for replay.
  • — Instrument per-layer gradient and activation distributions with sampling to control overhead.
  • — Test custom autograd operations against finite differences and framework reference implementations.
  • — Treat mixed precision, gradient accumulation, clipping, and distributed reduction order as part of the training algorithm.

Staff lens

  • — Build a failure tree that distinguishes data, forward, loss, backward, optimizer, and distributed synchronization.
  • — Require resumability tests that prove checkpoints restore the complete training state.
  • — Choose checkpointing and parallelism from measured memory and communication bottlenecks.

Interview defense

Explain backpropagation and why gradients accumulate.

Backpropagation is reverse-mode autodiff over a computation graph. Each operation computes a vector-Jacobian product from the output gradient and saved forward context. If a value reaches the loss through multiple paths, the chain rule sums every path contribution, so its gradient must accumulate before the optimizer updates parameters.

Expect the interviewer to press on

  • — Why is reverse-mode efficient for neural networks?
  • — How would you verify a custom backward function?
  • — What is the difference between vanishing gradients and a detached graph?

Misconceptions to remove

“The error is sent backward through the model.”

Derivative contributions are propagated through a recorded computation graph; the forward error value itself is not replayed backward.

“Backpropagation updates the weights.”

Backpropagation computes gradients. An optimizer and training loop decide how and when parameters change.

“A zero gradient proves the framework is broken.”

The derivative may correctly be zero because of saturation, masking, symmetry, or the local objective.

Check your model

1. Why does a shared parameter require gradient accumulation?

The total derivative is the sum of contributions from every downstream path connecting that parameter to the loss.

2. Why not build the full Jacobian for every operation?

It is usually far larger than needed. Reverse-mode propagates vector-Jacobian products directly and avoids materializing it.

Prove the mechanism

Add addition, tanh, and exponentiation to the engine, then gradient-check a branched expression at five random inputs.

Add a production constraint

Add tensor shapes and broadcasting rules. Write a failing test for a broadcasted gradient that must be reduced back to the input shape.

Artifact: Tiny autodiff engine and tests

courses/ai-engineering/reference-impl/backprop/local_autodiff.py

Download reference implementation

Primary references and next links

References

  1. 1. Learning representations by back-propagating errors

    Rumelhart, Hinton, and Williams. Influential 1986 neural-network learning paper.

  2. 2. Automatic differentiation package

    PyTorch. Official autograd documentation.

  3. 3. The Autodiff Cookbook

    JAX. Official guide to Jacobian products and differentiation modes.

Continue through the graph

Glossary: computation graph · chain rule · reverse mode · vector-Jacobian product · gradient accumulation