Build labFoundation75 min estimateOriginal publication

From Experiment to Service Boundary

An experiment proves that a path is worth studying. A service boundary states exactly which inputs, artifacts, configuration, outputs, and failures production is willing to support.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Do not deploy a notebook. Extract a small, deterministic inference contract: a versioned per-feature request schema with names, meanings, units, strict numeric types, bounds, and byte limits; validated and ordered preprocessing; a versioned response schema; and explicit error behavior. Derive one stable release identity from the model and preprocessing digests, complete input/output schemas, threshold and decision meanings, size limits, and configuration. Expose that identity in every response so two thresholds cannot masquerade as the same release. Load artifacts at startup, reject invalid input before model execution, and validate outputs before they cross the boundary.

Why this matters

Exploratory code often depends on cell order, ambient files, mutable globals, local credentials, implicit library versions, and manual cleanup. Copying that state into a handler produces a service that cannot explain which code and artifact answered a request or reproduce a failure.

You will be able to

Separate exploratory workflow from the supported inference contract.
Bind schemas, preprocessing, model identity, thresholds, decisions, and limits into one content-derived release identity.
Reject booleans, unknown semantics or units, out-of-range values, oversized requests, and invalid outputs before they cross the boundary.
Test deterministic boundary behavior without depending on an HTTP framework.
Design release, canary, observability, and rollback evidence around contract versions.

Prerequisite contract

— Functions, data classes, and unit tests
— Basic request/response APIs
— A reproducible experiment artifact

Your Vector Loop for this lab

01
Model
Represent serving as a content-identified transformation around model, preprocessing, schema, and decision configuration.
02
Derive
Derive input, configuration, output, and failure invariants from the product boundary.
03
Build
Implement a pure inference service with an injected predictor and immutable configuration.
04
Stress
Inject schema drift, shape errors, non-finite values, invalid scores, and hidden state.
05
Operate
Release artifact and contract versions together with telemetry, canary, and rollback evidence.
06
Defend
Explain what the boundary guarantees and which reliability concerns remain outside it.

Extract supported behavior from exploratory state

Experimental convenience	Production contract	Why the change matters
cell-created global	constructor dependency or immutable config	startup fails before traffic
ambient dataframe columns	versioned request schema	shape and meaning are validated
local model filename	content-addressed or versioned artifact identity	responses are attributable
manual cleanup	source-controlled preprocessing function	training and serving can share tests
printed score	versioned response schema	consumers know meaning and provenance
stack trace	classified boundary error	clients can retry, correct, or stop safely

The supported surface should be smaller than the experiment. Exploratory plots, search loops, ad hoc diagnostics, and one-off data repair may remain development tools. Promote only the transformation that the product needs to call repeatedly and the evidence required to operate it.

Derive four groups of invariants

01Request invariantsSchema version, exact feature names, declared semantics and units, strict non-boolean numeric types, ranges, encoding, maximum bytes, and identity or tenant context.
02Artifact invariantsModel and preprocessing content digests, tokenizer, label map, threshold, input/output schemas, decision meanings, size limits, dependency lock, and hardware compatibility must form one release identity.
03Response invariantsSchema and contract version, content-derived release ID, model and preprocessing versions, output range and units, decision meaning, and safe handling for abstention or partial output.
04Operational invariantsTimeout, cancellation, concurrency, idempotency, authentication, authorization, logging, privacy, and overload behavior surround pure inference.

Not every invariant belongs in the model function. The pure boundary below owns deterministic validation and attribution. A transport adapter should own parsing and status mapping; infrastructure should own deadline propagation, admission control, and isolation. Document the seam so failures do not fall between teams.

Build the inference contract before the web handler

inference_contract.py

1def format_example() -> str:
2    service = InferenceService(ILLUSTRATIVE_CONFIG, illustrative_predictor)
3    response = service.infer(ILLUSTRATIVE_REQUEST)
4    return json.dumps(asdict(response), sort_keys=True)
5 
6if __name__ == "__main__":
7    print(format_example())

Expected output

{"contract_version": "inference-contract-v1", "decision": "review", "model_version": "illustrative-model-v1", "preprocessing_version": "illustrative-preprocessing-v1", "release_id": "illustrative-release-v1@sha256:e9874ea4736a943301dedc24d8af7af3196d859c89026dae76be9198db1f81e2", "release_version": "illustrative-release-v1", "request_id": "req-001", "response_schema_version": "illustrative-response-v1", "score": 0.5, "score_semantic": "illustrative review score", "score_unit": "ratio"}

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/inference_contract.

The predictor is an explicitly illustrative deterministic fixture, not a trained model or benchmark. The response release_id is the release label plus a SHA-256 digest over model and preprocessing digests, feature names/semantics/units/bounds, request and response schema versions, output semantics/range, threshold and decision labels, and size limits. Changing only the threshold changes this identity. Injection keeps boundary tests independent of a large artifact while production supplies a predictor from the approved bundle.

Stress every source of hidden state

Injected fault	Boundary response	Evidence to retain
old request schema	reject before prediction	expected and received schema versions
missing or unknown feature	reject before prediction	expected and received feature names
boolean, NaN, or infinity	reject before prediction	strict type/finite validation and upstream version
wrong semantic, unit, or range	reject before prediction	feature-schema and preprocessing versions
oversized request or identifier	reject before prediction	declared byte/character limit
score outside supported range	fail response	artifact, config, and output validation
threshold changed under old label	derive a different release ID	complete release identity payload
missing artifact at startup	do not become ready	release identity and load error
deadline expires	cancel or discard result	deadline, phase timing, and capacity state
concurrent mutation	prevent by immutability or isolation	request IDs and config identity

Also test repeatability under a fresh process, dependency lock, and clean filesystem. If the same request changes after restarting with the same declared release, the contract is missing a source of randomness, state, remote dependency, or artifact identity.

Release the model as one component of a service

01BuildCreate an immutable release bundle containing source, dependency lock, schemas, transforms, model artifacts, configuration, and provenance.
02QualifyRun unit, contract, data, artifact-load, offline quality, security, and performance tests in a production-like environment.
03ShadowExercise representative traffic without changing user-visible decisions; compare outputs, errors, timing, and resource use.
04CanaryRoute an authorized scope to the new release with measurement gates, deadline behavior, and immediate rollback intact.
05ObserveRecord request schema, contract, model, transform, route, timing, validation outcome, and privacy-safe correlation IDs.
06RollbackRestore the previous compatible bundle and verify that clients, stored features, queues, and response consumers remain compatible.

Readiness should fail until the approved artifact loads and a local contract probe passes. Liveness should answer whether the process can continue, not whether the model is semantically good. Keep these health questions distinct from product-quality monitoring.

Operate at three altitudes

Production lens

— Load immutable artifacts and validate their compatibility before advertising readiness.
— Propagate deadlines and cancellation; do not let abandoned requests consume unbounded model capacity.
— Log the content-derived release ID and timing metadata without copying sensitive feature values into unrestricted telemetry.
— Keep old contract versions only with an explicit compatibility and retirement policy.

Staff lens

— Define ownership at the seams among data producers, model release, inference service, transport, and consuming product logic.
— Require one release identity to cover model, preprocessing, thresholds, schemas, dependencies, and configuration.
— Design rollback for data and client compatibility, not only container replacement.

Interview defense

How would you turn a successful notebook experiment into a production service?

I would extract reusable source components and define versioned request and response contracts around pinned model and preprocessing digests, label maps, thresholds, and configuration. I would derive a content-based release ID across that whole identity and return it with every response. Before inference I would validate exact feature names, meanings, units, strict numeric types, finite values, bounds, and request size; afterward I would validate output semantics and range. I would keep the core independent of HTTP, then add deadlines, admission, auth, observability, privacy controls, canary gates, and rollback around it.

Expect the interviewer to press on

— What belongs in readiness versus liveness?
— How do you prevent training-serving skew?
— What must remain compatible during rollback?

Misconceptions to remove

“Putting notebook code behind an API makes it production-ready.”

Transport does not remove hidden state, pin artifacts, define semantics, add validation, or create release and rollback evidence.

“The model file is the model version.”

Prediction meaning also depends on preprocessing, tokenizers, label maps, thresholds, input/output schemas, limits, dependencies, and configuration; bind all of them into the release identity.

“If inference returns without throwing, the response is valid.”

The output may be non-finite, out of range, stale, unauthorized, or incompatible with the declared response contract.

Check your model

1. Why inject the predictor into the inference boundary?

It separates model computation from request, configuration, and response semantics, making the boundary deterministic and testable with a small fixture.

2. Why should an artifact-load failure prevent readiness?

A process that cannot load the declared release cannot honor inference contracts; admitting traffic would convert a startup defect into request failures.

Prove the mechanism

Extract a versioned inference boundary from a small exploratory script. Remove ambient files and globals; define each feature's name, meaning, unit, type, and range; impose request-size limits; derive the release identity from the complete model/preprocessing/schema/threshold configuration; and prove invalid input never calls the predictor.

Add a production constraint

Add a transport adapter with deadline propagation, idempotent request IDs, structured error mapping, readiness checks, and a backward-compatible second request schema without weakening version checks.

Artifact: Inference contract

courses/ai-engineering/reference-impl/inference_contract/inference_contract.py

Download reference implementation

Primary references and next links

References

1. MLOps: Continuous delivery and automation pipelines in machine learning
Google Cloud Architecture Center. Official guidance on modular source components, validation, metadata, model serving, monitoring, and experimental-operational symmetry.
2. The ML Test Score
Breck et al.. Primary paper on tests and monitoring required around production ML behavior.
3. Hidden Technical Debt in Machine Learning Systems
Sculley et al.. Primary paper on the surrounding data, configuration, consumers, and system dependencies that dominate production ML risk.
4. Secure Software Development Framework
National Institute of Standards and Technology. Primary government guidance for release provenance, documented requirements, protected artifacts, and secure development decisions.

Continue through the graph

Design a Multi-Tenant AI Gateway →
Continue from one inference boundary to shared routing, isolation, and release policy.
Engineering baseline roadmap →
Connect serving to reproducibility, data contracts, measurement, and cost.

Glossary: inference contract · request schema · artifact identity · training-serving skew · readiness · canary · rollback