Design reviewFoundation55 min estimateOriginal publication

Draw the AI System Boundary

Before selecting a model or framework, draw the complete decision path and make every uncertain handoff, external dependency, owner, and actuator visible.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

An AI system boundary is an accountability map around an outcome, not a box around a model. Include input sources, deterministic transforms, probabilistic components, policy checks, people, external services, actuators, telemetry, and feedback. For every component and crossing, state its owner, input/output contract, failure mode, and evidence. The model may propose or score; the surrounding system still decides what is accepted, shown, stored, or acted upon.

Why this matters

If a team labels the model as the system, failures in identity, data freshness, retrieval, policy, UI, human review, or downstream action become invisible. The result is a model metric that looks healthy while the product makes the wrong decision—or a deterministic integration failure misdiagnosed as model quality.

You will be able to

Define an AI system around a product decision and operating context rather than a model endpoint.
Classify deterministic, probabilistic, data, human, and external components without treating any class as automatically safe.
Trace data, decisions, authority, and feedback across every boundary crossing.
Assign ownership, contracts, failure modes, and observable evidence before implementation.
Use the map to decide whether AI is necessary and where deterministic controls belong.

Prerequisite contract

— Competent programming in Python or TypeScript
— Basic request/response and data-flow concepts
Academy roadmap →

Your Vector Loop for this lab

01
Model
Frame one product outcome, its actors, and the authority to act.
02
Derive
Separate behavior types and trace every data and decision crossing.
03
Build
Encode the map as a validated, owner-bearing system manifest.
04
Stress
Remove owners, break contracts, shift data, and bypass human authority.
05
Operate
Version the boundary with telemetry, incident paths, and change review.
06
Defend
Explain why the model is only one component and where final authority lives.

Start at the consequence, then work backward

Choose a concrete consequence: rank refund requests, draft a response, block a transaction, or schedule an inspection. Identify who experiences the result and who is accountable for it. Then trace backward from the actuator to every value, policy, model output, source, and person that can change that result. This yields a system boundary that matches operational reality.

A useful boundary describes the complete decision path, not merely the inference call.
Question	What it exposes	Weak answer
What action can occur?	the actuator and blast radius	the model returns JSON
Who may authorize it?	human and policy authority	the workflow handles it
What information changes it?	data and semantic dependencies	the prompt contains context
What can fail independently?	components and contracts	the AI can be wrong
What evidence survives?	observability and audit path	we log the response

Separate components by behavior, not team chart

Class	Contract to write	Representative failure
deterministic	same valid input produces specified output	parser accepts malformed or stale data
probabilistic	distributional quality under named slices	scores shift or plausible output is wrong
data	schema, meaning, time, lineage, and use	correct type carries the wrong meaning
human	authority, information, timing, and escalation	automation bias or inconsistent judgment
external	dependency, policy, and failure envelope	provider changes behavior or becomes unavailable

These classes are diagnostic, not a safety ranking. Deterministic code can consistently enforce the wrong rule. A person can misunderstand the evidence or be unable to intervene in time. An external model may be technically reliable but violate a data-region requirement. Classifying behavior tells you what kind of contract and test to write.

product action = actuator(policy(validated model output, context, authority))

This is a dependency statement rather than a numeric equation. Each function is a separately owned boundary with distinct tests and failure behavior.

Map the crossings and reveal the gaps

AI system boundary mapper

Classify components by behavior, connect data and decisions, then reveal missing owners, contracts, and failure evidence.

Put control where it can be enforced

Inspect the request path, choose a system decision, then compare it with the boundary that owns validation, authorization, measurement, and approval.

Request scenario

Summarize a permitted support ticketIssue a refund with incomplete policy evidenceGrant production access from a chat request

Summarize a permitted support ticket

An authenticated support agent asks for a concise summary of a ticket they are already allowed to view. The system will not contact the customer.

Deterministic software
Validation
Validate the ticket ID and output schema, cap length, redact secrets, and reject malformed model output before it reaches the agent.
Data access
Authorization
Verify that this agent may read this ticket before retrieval. The model never decides access from prompt text.
Probabilistic model
Measurement
Measure factual consistency, source-span coverage, latency, and agent correction rate on this workload.
Human authority
Approval
No extra approval is needed to draft the summary, but the agent—not the model—decides what is sent to the customer.

Your boundary decision

Run behind deterministic gatesRequire explicit human approvalReject before the model or tool

Choose a decision. The request description is evidence; model confidence is not authorization.

Turn the diagram into an executable review artifact

A picture is easy to approve while ownership stays ambiguous. The reference artifact stores component kind, owner, contract, failure mode, evidence, purpose, and decision owner. Every crossing records payload, classification, owner, contract, failure behavior, authentication context, and evidence. Its validator rejects missing obligations, unknown endpoints, duplicate or self-referential flows, zero-flow maps, disconnected components, and maps with no probabilistic component that are labeled as AI systems.

ai_system_boundary.py

1if __name__ == "__main__":
2    print(boundary_report(EXAMPLE_BOUNDARY))

Expected output

boundary=refund-triage
components=6 flows=5
probabilistic_components=1
human_decision_points=1
external_flows=1

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/system_boundary.

01Name the outcomeUse a narrow verb and object, such as prioritize refund requests. Avoid labels like AI platform that conceal the decision.
02Inventory componentsInclude sources, transformations, stores, model calls, validators, policy, people, UI, actuators, telemetry, and external dependencies.
03Record every crossingName the payload, schema or contract, classification, authentication context, and behavior when the receiving side is unavailable.
04Assign evidenceFor each failure mode, identify the signal that distinguishes it from adjacent failures and the owner who responds.

Stress the boundary, not just the prompt

01Bypass the reviewerVerify that the model cannot directly invoke the refund actuator and that authorization is checked at execution time.
02Serve a stale policy snapshotConfirm that the model output carries policy and model versions and that stale combinations can be refused or isolated.
03Change the input distributionSlice quality and rejection behavior by language, request type, customer state, and data-source version rather than reporting one aggregate score.
04Lose the external providerExercise timeout, malformed output, quota, and partial-stream behavior. A declared queue, manual path, or refusal is safer than an unreviewed fallback.
05Poison the feedback pathCheck whether model-influenced human decisions are later treated as independent truth and whether rejected cases disappear from training data.

Keep the map synchronized with the running system

Version the boundary beside the release and require a diff when components, owners, contracts, purposes, or authority change.
Emit resolved model, prompt, policy, data, and boundary versions into trace evidence without logging sensitive content by default.
Give each crossing a failure behavior: reject, retry within a budget, queue, degrade explicitly, escalate, or stop.
Test the end-to-end product decision as well as component metrics; a healthy model can feed a broken actuator or misleading interface.
Review whether a deterministic rule or non-AI workflow could meet the requirement with lower uncertainty and operating cost.
Treat feedback collection, retraining, decommissioning, and retained data as components of the lifecycle boundary.

Operate at three altitudes

Production lens

— Trace one outcome across source, transformation, inference, policy, human, and actuator versions.
— Authenticate authority at the side-effect boundary instead of trusting a model or client assertion.
— Give external dependencies explicit timeout, quota, regional, data-use, and degradation contracts.
— Monitor product decisions and failures by component and slice; do not substitute provider uptime for system correctness.

Staff lens

— Use the boundary to assign ownership without forcing every component into one team or service.
— Separate rollback units so prompt, model, policy, data, UI, and actuator changes can be isolated.
— Challenge whether the intended use is narrow enough to measure and whether a non-AI alternative is preferable.
— Include impacted users, operators, domain owners, security, privacy, and legal constraints when the context requires them.

Interview defense

Draw the system boundary for an AI-assisted refund-triage product.

I would begin with the consequence: prioritization, with a support operator retaining refund authority. I would map authenticated requests, deterministic validation, versioned policy data, the probabilistic scorer, evidence shown to the operator, the authorization check, the idempotent refund service, telemetry, and feedback. Each component and flow gets an owner, contract, failure mode, classification, and version. I would prevent direct model access to the actuator, test provider and stale-data failures, evaluate product decisions by slice, and retain an explicit manual or refusal path.

Expect the interviewer to press on

— Where would you enforce authorization?
— How could human review still fail?
— What changes require a new boundary review?
— When would you remove the model from this design?

Misconceptions to remove

“The model endpoint is the AI system boundary.”

The system includes every source, policy, person, interface, external dependency, actuator, and feedback path that shapes or experiences the outcome.

“Deterministic components do not need risk analysis.”

They can reliably apply stale policy, accept malformed data, leak authority, or amplify a model error into a side effect.

“Adding a human in the loop makes the system safe.”

Oversight needs authority, time, comprehensible evidence, training, escalation, and measurement; a nominal click can become automation theater.

“A system diagram is enough documentation.”

A useful boundary also records purpose, ownership, contracts, versions, failure behavior, and observable evidence that can be reviewed and tested.

Check your model

1. Why should the downstream actuator appear inside an AI system boundary even when it contains no model code?

It determines the real consequence and must enforce authorization, idempotency, and failure behavior. Excluding it hides how a model output becomes an action.

2. What is the practical value of classifying a component as probabilistic rather than deterministic?

It changes the contract and evidence: behavior must be measured over a distribution and slices, while deterministic behavior can be tested against specified input/output invariants. Neither classification implies safety.

3. What makes a human-review step operationally real?

The reviewer has explicit authority, sufficient context, enough time, an escalation path, and measured oversight effectiveness before the action occurs.

Prove the mechanism

Map an AI feature you know as a versioned boundary manifest. Include at least five behavior classes or explain why a class is absent, name every data and decision crossing, and identify one observable failure for each component.

Add a production constraint

Add a policy-as-code check that rejects any path from a probabilistic component to a high-impact actuator unless an explicit validator and authority component appear on that path.

Artifact: AI system boundary map

courses/ai-engineering/reference-impl/system_boundary/ai_system_boundary.py

Download reference implementation

Primary references and next links

References

1. Artificial Intelligence Risk Management Framework (AI RMF 1.0)
National Institute of Standards and Technology. Primary framework for mapping system context, intended purpose, actors, components, risks, measurement, and human oversight across the lifecycle.
2. AI RMF Playbook: Map
NIST AI Resource Center. Official operational guidance for documenting purpose, requirements, component outputs, accountability, and human roles.
3. ISO/IEC/IEEE 42010:2022 — Architecture description
International Organization for Standardization. Official standard record distinguishing an entity's architecture from the architecture description used to express concerns, viewpoints, and relationships.

Continue through the graph

Academy roadmap →
Place the boundary practice in the complete capability sequence.
AI/ML interview map →
Connect component boundaries to system-design questions.
System design practice →
Apply ownership, contracts, and failure analysis to adjacent systems.

Glossary: system boundary · deterministic component · probabilistic component · actuator · human oversight · failure envelope