Reproducible Experiments, Not Reproducible Notebooks

A notebook records cells and outputs, but its kernel may also contain variables created by deleted or out-of-order cells, imported local modules that later changed, cached data, environment variables, working-directory assumptions, and package versions that never appear on the page. Restart-and-run-all is a valuable test, yet it still says nothing about whether another machine can reconstruct the same inputs.

Do not put wall-clock start time into the content identity: time would force equivalent inputs to receive different IDs. Record attempt ID, start time, operator, and infrastructure allocation in a separate execution record that points to the immutable manifest ID. One manifest may therefore have several attempts whose outputs can be compared.

The reference implementation hashes code and data bytes; recursively freezes canonical configuration, runtime, and hardware mappings; rejects wildcard, conditional, and range dependency specifiers; and binds the entry command, evaluation data/code/metrics/slices, named RNG seeds, worker policy, and deterministic-algorithm policy into identity. It rejects non-finite JSON and refuses to verify an execution when current bytes differ from the recorded digests.

1if __name__ == "__main__":
2    example = example_manifest()
3    print(f"run_id={example.run_id}")
4    print(f"code_sha256={example.code_sha256[:12]}")
5    print(f"data_sha256={example.data_sha256[:12]}")
6    print(f"inputs_verified={verify_inputs(example, code=EXAMPLE_CODE, data=EXAMPLE_DATA)}")

run_id=20b6ce0ee288
code_sha256=c2121bf12852
data_sha256=f2823c5bc5d7
inputs_verified=True

1. 01Resolve inputsConvert references such as latest, a mutable table, or a branch name into immutable revisions or snapshots before execution.
2. 02Canonicalize the manifestSerialize one validated schema with stable key ordering, finite values, explicit units, and no hidden defaults.
3. 03Execute from clean stateCreate a fresh process and environment, verify input digests, then invoke one entry command rather than depending on prior cells.
4. 04Retain evidenceStore the resolved manifest, stdout/stderr, metrics by slice, artifacts, exit status, and attempt metadata together.
5. 05Reconstruct independentlyHave another clean worker—or another engineer—resolve the manifest without access to the original interactive session.

Random libraries keep separate generators. Data-loader workers can receive their own seeds. GPU operations may select nondeterministic algorithms, parallel reductions can accumulate floating-point values in a different order, and library releases can change implementations. PyTorch explicitly states that completely reproducible results are not guaranteed across releases, commits, platforms, or CPU/GPU execution even with identical seeds.

1. 01Reorder notebook cellsRestart the kernel and execute the promoted command without notebook state. Any missing symbol or changed output exposes an undeclared dependency.
2. 02Mutate one data byteThe manifest must fail verification before training. A stable URI with changed contents is not the same input.
3. 03Relax one package pinResolve the environment twice and compare dependency graphs. A range expresses compatibility intent, not a historical execution identity.
4. 04Change device or precisionCompare whether the claim requires exact values, tolerance-based equivalence, or distributional bounds; do not silently accept the difference.
5. 05Rerun only the winning seedExpose seed selection in the report. Searching many seeds and reporting one is experimental selection, not a reproducibility control.

• Keep notebooks thin: import tested functions, render retained results, and avoid owning the only copy of training or evaluation logic.
• Resolve and verify manifests before allocating expensive compute so invalid inputs fail early.
• Build environments from lock files and retained artifacts; exact pins alone can still resolve unavailable or different distribution files.
• Separate manifest identity from execution-attempt metadata and never overwrite earlier result artifacts.
• Record evaluation dataset, slices, metric implementation, thresholds, uncertainty, and failures—not only the best aggregate number.
• Define retention, access, and deletion rules for data and artifacts; reproducibility does not authorize indefinite storage.
• Test reconstruction on a schedule before an incident or audit makes the missing dependency urgent.