Build labIntermediate105 min estimateOriginal publication

Convolutions and Locality as an Inductive Bias

A convolutional layer does not discover locality from nothing. Its connectivity, shared kernel, stride, dilation, and boundary rule declare which translations the model should treat alike.

Authorship: InterviewsVector
Published / updated: 2026-08-25 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

A 2D neural convolution is commonly implemented as cross-correlation: slide one shared kernel over local input neighborhoods, multiply corresponding values, and reduce them. For valid padding, output height is floor((H − dilation·(Kh−1) − 1)/stride)+1, with the analogous width formula. Shared weights make the operator translation-equivariant only under declared assumptions: the same kernel, compatible stride shift, layout and numeric convention, and a comparison domain whose receptive fields do not cross a changed boundary. Bind input and kernel identities and content, cap shapes and work, and report excluded boundary positions. Equivariance of one linear operator does not imply invariance of a classifier or semantic suitability for every grid.

Why this matters

Convolutions concentrate statistical and computational capacity on local repeated patterns. That can be an excellent prior for images, spectrograms, sensor arrays, or spatial fields—and a bad one when coordinates have unrelated meanings. Shape mistakes, implicit padding, layout swaps, or a stride-misaligned augmentation can invalidate the property the architecture was chosen to provide while every tensor remains numerically valid.

You will be able to

Implement bounded two-dimensional cross-correlation from explicit tensor indices.
Derive output shape from input, kernel, stride, dilation, and padding.
Explain weight sharing, local connectivity, translation equivariance, and task-level invariance separately.
Define the valid cropped comparison domain for an input translation and compatible output shift.
Audit layout, padding, stride, dilation, kernel, input, numeric, boundary, owner, identity, and workload contracts.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Represent the operator as local connectivity, one shared kernel, layout, stride, dilation, and boundary policy.
02
Derive
Derive valid output shape, receptive-field coordinates, work, and the translation mapping on the comparable domain.
03
Build
Implement pure-Python cross-correlation and a content-addressed equivariance suite with explicit exclusions.
04
Stress
Break shapes, identities, boundary assumptions, stride alignment, numeric safety, container immutability, and work bounds.
05
Operate
Validate production preprocessing, augmentation, layout, latency, memory, and slice behavior beside operator tests.
06
Defend
Defend why locality and weight sharing fit the domain and where equivariance or invariance claims stop.

Name the executed operator, not the family nickname

Y[i,j] = Σᵤ Σᵥ X[i·sᵧ + u·dᵧ, j·sₓ + v·dₓ] K[u,v]

This is valid 2D cross-correlation: the kernel indices are not flipped. Many deep-learning APIs call it convolution. The contract names the exact operation so a mathematical flip is not hidden behind terminology.

Choice	Inductive or systems effect	Common silent failure
local kernel	limits direct connectivity	kernel misses a domain-scale dependency
weight sharing	same detector is reused by position	coordinates are not actually exchangeable
stride	subsamples output locations	augmentation shift is not stride aligned
dilation	widens receptive field without dense weights	gridding skips important local structure
padding	defines artificial boundary context	edge behavior is presented as interior equivariance
layout	maps indices to height, width, and channels	NHWC and NCHW are silently swapped

Derive shape and cost before allocating output

Hout = ⌊(H + 2p − dᵧ(Kh−1) − 1)/sᵧ⌋ + 1

Width follows the same formula. The artifact deliberately supports p=0 valid padding and checks that the effective dilated kernel fits before arithmetic begins.

work = O(Hout · Wout · Kh · Kw) per single-channel case

Batches and input/output channels add corresponding multiplicative dimensions. Optimized libraries change constants and memory traffic, not the direct-algorithm index count.

A trustworthy implementation calculates effective kernel extent, output shape, and operation count from integers with hard global and contract bounds. It rejects a zero-sized domain, ragged rows, booleans masquerading as numbers, non-finite products, and sums outside the declared binary64 range before returning a plausible matrix.

Prove the translation property on the domain where it applies

Cₖ(TΔx)[i+Δᵧ/sᵧ, j+Δₓ/sₓ] = Cₖ(x)[i,j]

The equality requires one shared kernel, a shift divisible by stride, and indices whose translated receptive fields stay in the valid output. Positions lost or created at the boundary are excluded and counted.

01Compute the referenceApply the named kernel to the original input under the bound layout, stride, dilation, padding, and arithmetic convention.
02Translate the inputShift it with explicit zero fill; do not wrap pixels from one edge to the other.
03Map output coordinatesDivide only stride-aligned translations and retain pairs whose source and destination outputs both exist.
04Report exclusionsCount the boundary outputs not compared so a small interior test cannot masquerade as full-image invariance.

Build a bounded operator and assumption audit

convolution_equivariance_audit.py

1    for out_y in range(out_height):
2        row: list[float] = []
3        for out_x in range(out_width):
4            terms: list[float] = []
5            for kernel_y in range(contract.kernel_height):
6                for kernel_x in range(contract.kernel_width):
7                    input_y = out_y * contract.stride_y + kernel_y * contract.dilation_y
8                    input_x = out_x * contract.stride_x + kernel_x * contract.dilation_x
9                    term = checked_input[input_y][input_x] * checked_kernel[kernel_y][kernel_x]
10                    if not math.isfinite(term):
11                        raise OverflowError("cross-correlation product overflowed")
12                    terms.append(term)
13            value = math.fsum(terms)
14            if not math.isfinite(value) or abs(value) > contract.maximum_absolute_output:
15                raise OverflowError("cross-correlation output exceeds the numeric contract")
16            row.append(value)

Expected output

example=illustrative_only
contract_version=equivariance-v1
evidence=equivariance-suite-001
evidence_content_id=equivariance-evidence@sha256:0a3358db122981ea839026357fedce97b72cfd705d6bfc565f6f8abaf8e9c4ab
kernel_content_id=convolution-kernel@sha256:4dac2ac8a98d9ddbe30cf09c3c65334f88c3a7229a8186eaf049da0bcebf01dc
output_shape=4x4
compared=9
excluded_boundary=7
maximum_error=0.000
status=EQUIVARIANT
assumption=WEIGHT_SHARING_VALID_DOMAIN_ONLY

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/convolution_equivariance

The excerpt is literal source. Capture copies exact tuple matrices and binds case, input, kernel, operation, layout, padding, stride, dilation, translation, shape, tolerance, numeric, and owner identities. The immutable contract carries the canonical kernel matrix and derives a full kernel content ID from its identity, shape, values, and numeric convention; every case and report retain that digest, and the public operator rejects a distinct matrix reused under the same friendly kernel ID and version. The audit revalidates constructor-bypass mutations, rejects duplicate case and input identities, recomputes the full suite hash, enforces shape and operation budgets across both passes, and returns unsupported when shift or crop assumptions do not authorize a comparison. Stable summation reduces avoidable error; bounded terms and outputs keep it from becoming an overflow oracle.

Stress boundaries before celebrating symmetry

01Move by one under stride twoReturn unsupported because no integer output translation represents the input shift.
02Compare the padded edgeShow that newly introduced zeros change receptive fields; keep those outputs outside the valid claim.
03Change the kernel versionReject the case before comparing values. Equivariance requires weight sharing, not merely equal shapes.
04Forge the evidence IDChange one matrix entry under the same display ID and prove the full content identity changes.
05Attack arithmetic and workReject ragged or mutable matrices, booleans, NaN, infinity, subnormal inputs, oversized values, outputs, shapes, cases, and operation counts.

Operate the complete spatial pipeline

Boundary	Release evidence	Failure to watch
decode and resize	golden pixels and geometry	orientation, crop, or interpolation shift
layout and normalization	shape, channel, scale, and dtype assertions	NHWC/NCHW or color-order swap
augmentation	seeded transform replay	labels invalid after transform
operator	reference values, gradients, and equivariance domain	padding or stride mismatch
task head	slice quality and calibration	operator symmetry does not serve product utility
runtime	latency, memory, kernel selection, determinism	fast path changes numeric behavior

Keep a tiny reference implementation because it explains indexing and catches contract movement, then compare the production kernel against framework and device-specific golden cases. The reference is intentionally not a performance implementation. Release decisions need multi-channel, batched, backward, preprocessing, hardware, and outcome evidence beyond this single-channel audit.

Operate at three altitudes

Production lens

— Version decode, resize, crop, channel order, normalization, layout, padding, stride, dilation, weights, precision, and runtime kernel together.
— Retain small reference cases for forward values, gradients, shape, and bounded equivariance comparisons across devices.
— Measure latency and memory on real shape distributions; direct operation counts do not predict optimized-kernel throughput alone.
— Evaluate boundary, scale, orientation, subgroup, and corrupted-input slices beside aggregate task quality.

Staff lens

— Require teams to state why locality and translation structure match the data-generating process.
— Define one tensor-layout and padding vocabulary across training, export, and serving systems.
— Separate operator correctness, architectural prior, augmentation policy, and product-invariance claims in review gates.
— Own golden reference cases across framework upgrades, accelerator kernels, quantization, and model conversion.

Interview defense

Implement a 2D convolution and explain when it is translation-equivariant.

I would first name cross-correlation versus flipped-kernel convolution, then derive output shape from input, effective dilated kernel, padding, and stride. The direct loop maps each output coordinate to a local input footprint and applies the same kernel, costing O(Hout·Wout·Kh·Kw) per channel pair. Weight sharing gives translation equivariance only for compatible stride shifts and a domain whose receptive fields preserve the same boundary context. I would bind layout and numeric conventions, kernel and input content, cap work, compare only valid mapped points, report exclusions, and avoid calling operator equivariance classifier invariance.

Expect the interviewer to press on

— Why do many libraries implement cross-correlation under the convolution name?
— How do stride and padding restrict equivariance?
— When is locality a harmful inductive bias?

Misconceptions to remove

“Convolution means every output is invariant to translation.”

The feature map is ideally equivariant on an authorized domain: it moves with the input. Invariance is a downstream property.

“Same padding preserves every spatial property at the edge.”

It preserves a chosen output size by inventing boundary context. Edge receptive fields differ from interior ones.

“A lower operation count guarantees a faster layer.”

Memory layout, device kernels, tensor shapes, fusion, launch overhead, and precision determine realized performance.

Check your model

1. Why must an input translation be divisible by stride in this audit?

The translated feature location must land on an integer output coordinate. Otherwise the discrete sampled output has no exact corresponding shift.

2. Why report excluded boundary points?

Their receptive-field context changed or left the valid domain, so including them would test a different assumption and hiding them would overstate coverage.

3. What property does weight sharing establish?

It applies the same local linear rule at each eligible location. Under compatible layout and boundary assumptions, that supports translation equivariance—not semantic usefulness or task invariance.

Prove the mechanism

Implement single-channel 2D cross-correlation with explicit layout, valid padding, stride, dilation, kernel and input identities, binary64 policy, shape and work caps. Add a translation audit that counts compared and excluded points. Test known values, dilation, stride misalignment, boundary exhaustion, stale kernels, duplicate identities, mutated evidence, booleans, NaN, infinity, subnormal and excessive values, ragged rows, and operation overflow.

Add a production constraint

Extend the contract to batched multi-channel inputs and grouped kernels. Derive parameter, multiply-add, activation-memory, and output-shape costs; add an independent gradient check; and compare an optimized framework operator to the reference across layout and padding variants without importing framework behavior into the oracle.

Artifact: Convolution equivariance audit

courses/ai-engineering/reference-impl/convolution_equivariance/convolution_equivariance_audit.py

Download reference implementation

Primary references and next links

References

1. Group Equivariant Convolutional Networks
Cohen and Welling. Primary paper used to check translation equivariance, invariance, and shared transformed-filter terminology.
2. A Guide to Convolution Arithmetic for Deep Learning
Dumoulin and Visin. Primary technical report grounding output-shape arithmetic for kernel extent, stride, padding, and dilation.
3. Conv2d
PyTorch. Official API reference used to verify that the common neural operator is cross-correlation and to check declared parameters.

Continue through the graph

Matrices as Small Programs →
Read the local linear operator as explicit index transformations.
Complexity for Tensor Programs →
Extend the direct loop into shape, compute, and memory budgets.
Normalization and Residual Paths →
Trace scale and gradient behavior after composing spatial operators into deep blocks.

Glossary: cross-correlation · kernel · weight sharing · locality · translation equivariance · invariance · stride · dilation · valid padding · receptive field