Build labFoundation75 min estimateOriginal publication

A First AI Workload Cost Model

Architecture becomes negotiable when demand and resource use share units. Build a transparent equation first; replace assumptions with measured distributions as evidence arrives.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Model cost as demand multiplied by resource consumption multiplied by explicit unit rates. Keep input tokens, output tokens, accelerator time, retained storage, and network movement separate because architectures change them differently. Label the workload window, request-shape assumptions, cache and retry behavior, retention policy, rate source, currency, and effective date. Then calculate both technical unit cost and cost per useful outcome. A first model is a conditional equation—not a vendor quote, benchmark, or promise.

Why this matters

A team that estimates only requests per second can miss prompt growth, long outputs, idle accelerators, duplicated retries, accumulating traces, or cross-region transfer. Those drivers are often locked in by architecture before a billing dashboard can reveal them.

You will be able to

Translate a workload into token, compute-time, storage, and network quantities with explicit units.
Keep illustrative rates distinct from observed usage and vendor billing data.
Calculate total cost, cost per request, and cost per useful outcome without unit errors.
Stress the estimate with tails, retries, retention, utilization, and changing demand.
Use sensitivity analysis to identify which assumption deserves measurement first.

Prerequisite contract

— Multiplication, rates, and unit conversion
— Requests, tokens, bytes, seconds, and GiB
— A defined workload boundary

Your Vector Loop for this lab

01
Model
Represent cost as demand, resource consumption, and explicit rate tables.
02
Derive
Derive token, accelerator-hour, storage, network, and unit-cost equations dimensionally.
03
Build
Implement a Decimal-based calculator with named illustrative assumptions.
04
Stress
Vary tails, retries, utilization, caching, retention, and transfer topology.
05
Operate
Reconcile estimates with usage records and invoices while preserving model versions.
06
Defend
Defend architecture using sensitivity and outcome economics rather than one total.

Draw the resource ledger before choosing infrastructure

Driver	Quantity	Typical architectural lever	Common omission
Input	requests × input tokens	prompt design, retrieval, prefix reuse	tool and retrieved context
Output	requests × output tokens	stopping, structured output, workflow design	retries and rejected generations
Compute	requests × occupied accelerator seconds	model, batching, utilization, placement	idle and warm capacity
Storage	bytes retained over time	retention, compression, sampling	embeddings, traces, indexes, checkpoints
Network	bytes crossing billed boundaries	co-location, caching, topology	replication and cross-region movement

Start with a business workload unit such as reviewed case, accepted suggestion, or completed task. Requests and tokens are resource units; they do not say whether the product delivered value. Track both so optimization cannot reduce token cost while making users retry more often.

Derive the first-order equations with units attached

R = requests/day × days; Tᵢₙ = R × mean input tokens/request; Tₒᵤₜ = R × mean output tokens/request

The mean makes this a first estimate. Replace it with request-shape buckets or a trace replay when tails, modes, or tenant mix matter.

Hₐcc = R × accelerator seconds/request ÷ 3,600 seconds/hour

Occupied accelerator time is not the same as provisioned time. Self-hosted capacity must also model utilization, replicas, headroom, and idle periods.

C = (Tᵢₙ/10⁶)rᵢₙ + (Tₒᵤₜ/10⁶)rₒᵤₜ + Hₐccrₐcc + GᵢBₛrₛ + GᵢBₙrₙ

Each term multiplies a consumed quantity by a matching rate. Currency, billing unit, discount basis, and effective date belong to the rate-table version.

Result	Formula	Decision use
technical unit cost	total variable cost / requests	compare implementation efficiency
outcome unit cost	total relevant cost / successful outcomes	compare delivered product value
marginal cost	change in cost / change in demand	forecast the next unit
sensitivity	change in total / change in one input	prioritize measurement and optimization

AI workload cost explorer

Change explicit illustrative demand, usage, and unit-rate inputs. Inspect which cost driver moves, then compare technical unit cost with cost per useful outcome.

Turn a workload into a capacity envelope

Start with demand and service assumptions. The output is a planning baseline to test with real traces—not a provider quote or a capacity guarantee.

Original requests / minuteMean input tokens / callMean output tokens / callRetried requests (%)Peak traffic multiplierMean occupied service time (s)Illustrative input rate ($ / 1M)Illustrative output rate ($ / 1M)

Cost rates are editable exercise inputs in USD per million tokens. The defaults are illustrative and are not current prices from any provider. Retry rate assumes each retried request makes one additional call. The peak multiplier changes instantaneous capacity, not daily volume.

Peak token throughput: 38,016 tokens/s; Includes peak multiplier and retry calls.
Daily token volume: 1,492,992,000 tokens; 1,119,744,000 input · 373,248,000 output
Peak concurrency estimate: 28.5 requests; Peak arrival rate × occupied service time.
Illustrative daily cost: $1,119.74; Editable input and output rates; excludes other infrastructure.

Read the sensitivity

Retries add 110,592,000 tokens/day and $82.94/day under these assumptions.
One additional second of occupied time adds about 23.8 peak concurrent requests; token demand stays unchanged.
Each $1.00 / 1M rate change moves the daily estimate by $1,119.74 for input or $373.25 for output.

Build a unit-safe calculator with explicit illustrative inputs

workload_cost.py

1# These are explicit illustrative inputs, not prices, benchmarks, or forecasts.
2ILLUSTRATIVE_WORKLOAD = WorkloadAssumptions(
3    days=30,
4    requests_per_day=10_000,
5    mean_input_tokens=D("800"),
6    mean_output_tokens=D("200"),
7    accelerator_seconds_per_request=D("0.4"),
8    stored_bytes_per_request=D("2048"),
9    egress_bytes_per_request=D("4096"),
10)
11 
12ILLUSTRATIVE_RATES = UnitRates(
13    input_usd_per_million_tokens=D("1"),
14    output_usd_per_million_tokens=D("4"),
15    accelerator_usd_per_hour=D("3"),
16    storage_usd_per_gib_month=D("0.10"),
17    egress_usd_per_gib=D("0.05"),
18)
19 
20print(format_example())

Expected output

example=illustrative_inputs_only
requests=300,000
input_tokens=240,000,000
output_tokens=60,000,000
accelerator_hours=33.33
storage_gib=0.57
egress_gib=1.14
total_usd=580.11
usd_per_request=0.001934

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/workload_cost.

Every displayed number is generated from deliberately invented inputs so the arithmetic can be tested. Substitute measured workload distributions and contract or billing rates before making a real decision. The storage term intentionally charges end-of-period volume for one month; a real retention model should integrate byte-hours or daily snapshots.

Stress the assumptions that architecture amplifies

01Replay request shapesSplit by prompt length, output cap, workflow, tenant, cacheability, and tool use. Averages erase expensive modes and queueing behavior.
02Add retry amplificationTimeouts, invalid structured output, provider retries, and user resubmission can multiply demand. Count every attempted call, not only successful product requests.
03Separate occupied from provisioned computeFor owned capacity, model replicas, headroom, maintenance, minimum scale, and utilization. Accelerator seconds alone describe work, not the entire capacity bill.
04Integrate retentionEmbeddings, logs, prompts, outputs, and checkpoints have different growth and deletion policies. End-of-month bytes are not byte-hours.
05Change topologyCross-zone, cross-region, internet, and provider boundaries may meter different movements. Draw source and destination before applying a rate.

Replace assumptions with observed cost and usage evidence

Model field	Observed source	Reconciliation check
requests and outcomes	product events	deduplicate retries and define success
input/output tokens	provider or gateway usage	match tokenizer and billable fields
accelerator time	scheduler and device telemetry	separate occupied and provisioned
retained bytes	storage inventory over time	apply deletion and replication
network movement	flow or billing records	preserve source, destination, and class
effective cost	normalized invoice data	include credits and commitments consistently

Version every estimate with its workload snapshot, architecture, rate table, currency, and date. Compare predicted and observed quantities before comparing money: usage mismatch points to workload or instrumentation; rate mismatch points to contract, allocation, or billing semantics.

Operate at three altitudes

Production lens

— Record token usage, attempted calls, cache status, retries, model route, and outcome IDs at the same gateway boundary.
— Separate variable, allocated shared, and fixed capacity costs instead of silently mixing them.
— Alert on unit-cost and demand-shape changes, not only monthly total spend.
— Reconcile modeled quantities with normalized billing records on a regular cadence.

Staff lens

— Use sensitivity to decide whether prompt length, output length, retries, utilization, retention, or network topology deserves the next engineering investment.
— Compare architecture alternatives under the same workload and reliability constraints.
— Keep rate negotiation and technical efficiency separate so a discount does not hide resource waste.

Interview defense

How would you estimate the cost of a new AI workload before launch?

I would define the useful workload unit and a request-shape distribution, then model input tokens, output tokens, accelerator or provider consumption, retained storage over time, and network movement separately. I would apply versioned unit rates with explicit currency and date, include retries, caching, utilization, replicas, and headroom where relevant, and report total, cost per request, and cost per successful outcome. I would run sensitivities, validate with a load test, and reconcile observed usage and billing after canary.

Expect the interviewer to press on

— How would self-hosted capacity change the model?
— Why can caching increase rather than decrease total cost?
— Which assumption would you measure first?

Misconceptions to remove

“Token price is the cost of an AI system.”

It omits compute placement, retries, storage, network, surrounding services, human review, and whether the request produced a useful outcome.

“Average tokens per request are enough for capacity and cost.”

Averages conceal request modes, correlated tails, burst concurrency, and long outputs that can dominate capacity and latency.

“A detailed spreadsheet is an accurate forecast.”

Detail only exposes assumptions. Accuracy comes from representative workload evidence, correct rates, reconciliation, and uncertainty ranges.

Check your model

1. Why keep input and output token costs separate?

They may have different rates and different architectural controls; combining them hides which change moved cost.

2. What does a gap between predicted token quantities and the invoice suggest?

First investigate workload boundaries, retries, instrumentation, tokenizer or billable-field semantics, and unmodeled routes before blaming the unit rate.

Prove the mechanism

Create three request-shape buckets for an AI workload and estimate token, accelerator-time, storage, and network quantities. Use explicit illustrative rates, show the dominant sensitivity, and state what evidence would replace each assumption.

Add a production constraint

Add retries, cache hit/miss paths, provisioned-capacity utilization, byte-hour retention, and cost per successful outcome. Reconcile a synthetic usage record without changing the original assumptions silently.

Artifact: AI workload cost model

courses/ai-engineering/reference-impl/workload_cost/workload_cost.py

Download reference implementation

Primary references and next links

References

1. Unit Economics
FinOps Foundation. Official guidance on resource-efficiency and business-outcome unit metrics, including cost per token and cost per outcome.
2. FOCUS Specification v1.3
FinOps Open Cost and Usage Specification. Primary specification for normalized cost, consumed quantities, units, and pricing semantics.
3. Optimize resource usage
Google Cloud Well-Architected Framework. Official guidance on workload requirements, load patterns, profiling, forecasting, and cost drivers.

Continue through the graph

Design a Multi-Tenant AI Gateway →
Place usage attribution, budgets, routing, and capacity controls at a production boundary.
Engineering baseline roadmap →
Connect cost modeling to measurement and reproducible experiments.

Glossary: unit economics · workload shape · unit rate · accelerator hour · GiB-month · egress · sensitivity analysis