Build labIntermediate90 min estimateOriginal publication

Computation Graphs Make Learning Inspectable

A forward graph is an executable dependency contract: every value has an owner, operator, input edge, order, and reproducible trace before gradients enter the story.

Authorship: InterviewsVector
Published / updated: 2026-08-25 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Represent a numerical program as a directed acyclic graph whose nodes are typed inputs, constants, and operator applications. Ordered edges encode operand dependencies—including repeated operands such as x + x—and a declared topological order guarantees every parent is available before its consumer. A trustworthy forward trace binds graph, operator semantics, model revision, scope, input source, owner, every intermediate value, and declared outputs. Recompute rather than trust the trace, reject cycles, unreachable nodes, wrong arity, incomplete inputs, reordered evidence, and content reuse. This lesson owns forward-value integrity; reverse-mode vector-Jacobian products belong to the next backpropagation lesson.

Why this matters

Tensor frameworks make graph construction feel invisible until a shape, mutation, branch, overflow, or stale trace breaks training. An explicit forward graph lets an engineer isolate whether the wrong result came from topology, operator semantics, bound inputs, evaluation order, or arithmetic before blaming optimization.

You will be able to

Translate a scalar expression into typed nodes and dependency edges.
Derive and validate a topological evaluation order for a directed acyclic graph.
Distinguish graph structure, operator contracts, concrete input bindings, and forward traces.
Build a validator that recomputes every node and content-addresses material evidence.
Define the forward contract that reverse-mode autodiff will later consume.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Name nodes, values, operators, directed edges, outputs, and graph identities.
02
Derive
Derive a topological order and the invariants that make forward evaluation well-defined.
03
Build
Implement typed nodes, exact arity rules, bound inputs, and a recomputed trace.
04
Stress
Inject cycles, dead nodes, duplicates, stale revisions, wrong values, and numeric overflow.
05
Operate
Persist graph and trace identities beside outputs so incidents can replay the exact forward path.
06
Defend
Explain what forward integrity establishes and what derivative correctness still requires.

Model a numerical program as a DAG

Start with value ownership. Input nodes receive values from a named evidence source. Constant nodes bind parameter-like literals to the graph revision. Operator nodes name parents rather than copying their values. Declared output nodes define the observable result. The graph is directed because information flows from parent to consumer, and acyclic because this bounded evaluator has no recurrent state or fixed-point semantics.

score = ReLU((usage × weight) + bias)

One expression becomes six nodes: usage, weight, bias, multiply, add, and ReLU. Naming intermediates turns an opaque answer into a replayable trace.

Graph element	Contract	Rejected ambiguity
node ID	unique and stable within revision	two values share one name
operator	versioned name and exact arity	add silently changes meaning
edge	parent must be declared and earlier	missing input or cycle
order	contains every node exactly once	partial or nondeterministic evaluation
output	declared reachable node	dead work is treated as evidence

Derive order, acyclicity, and reachability

∀(u → v) ∈ E: position(u) < position(v)

If every edge points from an earlier position to a later one, the declared order supports forward evaluation and rules out a directed cycle under this complete node set.

A complete order must contain exactly the declared nodes. That prevents a validator from proving only a convenient subgraph. Reach backward from every declared output through its parent edges; if any declared node is absent from that closure, it is dead evidence and the contract rejects it. This strict convention makes graph revisions deliberate. A production framework may retain diagnostics or auxiliary outputs, but they should be declared rather than silently ignored.

01Index IDsReject duplicate nodes and duplicate positions before resolving any edge.
02Check local operator contractsInputs and constants require zero operands, ReLU one, and add or multiply two under this version; repeated operand IDs preserve valid multiplicity.
03Prove the global orderEvery parent must exist and precede its consumer; every node must reach an output.

Build and verify the forward trace

Evidence supplies exactly the declared input nodes and claims one value for every node in topological order. The validator requires exact claims for bound inputs and constants, recomputes operators, and applies the contract's immutable absolute tolerance only to derived values. It rejects unsupported numeric values, overflow, and derived underflow before hashing the complete graph and trace payload. A friendly trace ID remains useful for humans but cannot stand in for content identity.

computation_graph_trace.py

1def format_example() -> str:
2    audit = validate_forward_trace(ILLUSTRATIVE_GRAPH, ILLUSTRATIVE_TRACE)
3    return "\n".join((
4        "example=illustrative_only", f"graph_version={ILLUSTRATIVE_GRAPH.graph_version}",
5        f"trace_id={audit.trace_content_id}", f"nodes={audit.node_count}",
6        f"score={audit.output_values[0].value:.3f}", f"decision={audit.decision}",
7    ))

Expected output

example=illustrative_only
graph_version=renewal-score-graph-v1
trace_id=sha256:9af034df543e6f630ee27897a0dfccb7cc370f1a2cf0eab196f1aca6091d5504
nodes=6
score=1.100
decision=PASS

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/computation_graph

Tamper	Why a friendly ID misses it	Validator response
change one constant	graph name can be reused	content identity changes
change one input	trace name can be reused	content identity changes
claim a wrong intermediate	final output might coincide	recomputation rejects node
reorder trace rows	same row set remains	declared-order check rejects
add a dead node	output stays unchanged	reachability check rejects

Operate the forward contract before adding gradients

Reverse-mode autodiff needs a valid forward graph, saved primal values, and a local derivative rule for each operator. If a forward trace is stale or an operator changed semantics, even a mathematically correct vector-Jacobian product belongs to the wrong program. Make forward replay a prerequisite for gradient checking rather than combining topology and derivative failures in one mystery.

Incident symptom	First forward question	Then investigate
wrong score	which first node diverges?	input, constant, or operator owner
NaN downstream	which first node becomes non-finite?	range and numeric policy
gradient mismatch	does the primal trace reproduce?	local derivative contract
run cannot replay	are graph and source revisions bound?	lineage and retention

Operate at three altitudes

Production lens

— Version graph topology, constants, operator semantics, declared order, and outputs as one executable contract.
— Bind input source, model, scope, owner, observation time, and every claimed intermediate to trace evidence.
— Recompute traces and stop at the first divergence instead of accepting final-output coincidence.
— Extend scalar contracts with tensor shape, dtype, device, broadcasting, and mutation rules before production use.

Staff lens

— Standardize graph and trace identities across training, serving, compilation, and incident systems.
— Make operator-semantic changes explicit compatibility events with replay tests over retained traces.
— Separate topology, forward value, derivative, and optimizer ownership so failures route to the responsible layer.

Interview defense

How would you represent and debug a small neural computation without relying on a framework?

I would create unique typed nodes for inputs, constants, and operations, exact arity contracts, explicit edges, declared outputs, and a complete topological order. I would prove each parent precedes its consumer, reject cycles and unreachable nodes, bind all inputs and revisions, then recompute every intermediate in order and content-address the graph plus trace. The earliest mismatched node localizes a forward failure. Only after the primal trace reproduces would I add local derivative rules and reverse-mode traversal.

Expect the interviewer to press on

— How does topological order prove evaluability?
— Why reject unreachable nodes?
— What extra contracts do tensors require?
— What must be saved for reverse-mode autodiff?

Misconceptions to remove

“A final output match proves the graph trace is correct.”

Different intermediate errors can cancel. Recompute and compare every declared node under the exact graph revision.

“Listing nodes is enough to define a computation graph.”

Edges, operator semantics, constants, order, outputs, scope, and revision determine executable meaning.

“Autodiff removes the need to understand the forward graph.”

Autodiff applies local rules to a forward program. Wrong topology or stale primal values yield gradients for the wrong computation.

Check your model

1. What invariant must every edge satisfy in a topological order?

The parent node must appear before the consumer node.

2. Why bind operator-contract version separately from graph version?

The same operator name can change semantics. Both topology and the meaning of each operation are required for replay.

3. Why does the validator require a complete claimed trace?

It makes every intermediate inspectable and prevents a partial proof from being presented as complete forward evidence.

Prove the mechanism

Add subtraction, division, and a stable sigmoid to the operator contract. Specify arity and domain rules, reject near-zero divisors under a versioned numeric policy, and add replay tests for every failure.

Add a production constraint

Extend nodes with tensor shape and dtype declarations, implement explicit broadcasting checks, and prove a trace remains compatible across a graph serialization round trip.

Artifact: Computation graph trace validator

courses/ai-engineering/reference-impl/computation_graph/computation_graph_trace.py

Download reference implementation

Primary references and next links

References

1. Learning long-term dependencies with gradient descent is difficult
Bengio, Simard, and Frasconi. Primary paper connecting composed dynamical computations with gradient propagation difficulties.
2. Automatic Differentiation in Machine Learning: a Survey
Baydin, Pearlmutter, Radul, and Siskind. Primary survey defining computational graphs, forward evaluation, and automatic-differentiation modes.
3. math.fsum
Python documentation. Official documentation for accurate floating-point summation used by the validator.

Continue through the graph

Backpropagation as Local Contracts →
Add reverse-mode derivative contracts after forward integrity is established.
A Neuron Is a Parameterized Decision Surface →
See how one affine activation becomes a graph of explicit values.
Academy roadmap →
Place forward graphs in the broader neural-training sequence.

Glossary: computation graph · directed acyclic graph · topological order · arity · reachability · forward trace · primal value