The Cost–Quality–Latency Operating Envelope

Start with a successful user journey, not a model call. Quality may include task completion, factual support, schema validity, policy compliance, calibrated abstention, and human acceptance. Latency may include queueing, time to first token, tool round trips, completion, and end-to-end user time. Cost includes model and infrastructure use plus retrieval, tools, retries, failed requests, human review, observability, reserved capacity, and incident load. Name which component is measured, where, over which population and window, and which outcome makes the spend worthwhile.

Evaluate candidate configurations on the same representative workload and report a vector rather than a magic score. A candidate is Pareto-dominated when another candidate is no worse on every declared dimension and better on at least one. The frontier narrows the discussion, but product and risk owners still choose constraints and the objective. Slice the evidence by user, language, domain, tenant, risk, prompt and output length, tool path, cache state, region, load, and failure mode where behavior or consequence differs.

1. 01Freeze the candidate identityBind model, weights, prompt, retrieval corpus and parameters, tools, policy, decoding, cache, runtime, hardware, region, and scheduler.
2. 02Measure comparable populationsUse representative task and traffic distributions with protected holdouts, minimum slice samples, uncertainty, and separately reported failures and refusals.
3. 03Remove dominated candidatesDiscard configurations that consume at least as much cost and latency while producing no better quality, reliability, or capacity under the declared evidence.
4. 04Apply hard constraintsGate critical slices, tail latency, reliability, unit economics, portfolio budget, telemetry, and operational readiness independently.
5. 05Canary the bounded profileObserve user-centered online signals, demand mix, fallbacks, spend, and capacity with rapid rollback; do not extrapolate a lab frontier indefinitely.

1def gate_operating_envelope(contract, evidence):
2    contract = validate_record(contract, OperatingEnvelopeContract)
3    evidence = validate_record(evidence, OperatingEnvelopeEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another operating-envelope contract")
6    # Cross-multiply exact quality and per-success cost counts,
7    # gate every slice, then forecast monthly cost from integer microunits.

example=illustrative_only
decision=WITHIN_OPERATING_ENVELOPE
slices=batch,interactive,safety-boundary
monthly_cost_units=41000
violations=
claim=LOCAL_ENVELOPE_GATE_NOT_SLO_OR_FINANCIAL_CERTIFICATION

The fixture gives interactive, batch, and safety-boundary workloads separate quality floors, p95 latency ceilings, per-request and per-success cost walls, sample floors, and observation-window minimums. Evidence supplies exact quality numerators and denominators, successful and attempted outcome counts, latency, integer microunit cost, monthly demand, freshness, and digest identity. The gate cross-multiplies outcome counts so retries can fail economics without float rounding, checks aggregate telemetry coverage, and rounds forecast microunits upward to whole invented cost units. It runs no model, benchmark, forecast, statistical interval, or billing query.

Define controls before a breach. Admission should use estimated work and current capacity, not only request count. Routing should choose among configurations already qualified for that workload and policy. Degradation should be explicit—queue, limit features, use a tested smaller model, provide a cached bounded answer, ask for clarification, escalate, or refuse—rather than silently lowering quality. Retry ownership and budgets must span clients, gateways, tools, and providers so a dependency failure does not multiply demand.

• Measure client-visible latency distributions, successful task outcomes, refusals, schema failures, safety slices, retries, fallback, queueing, and abandonment by workload and version.
• Reconcile token, accelerator, provider, retrieval, tool, network, storage, telemetry, and review costs to successful outcomes and responsible organizational units.
• Compare actual demand and mix with the forecast; volume, length, tenant, route, and cache shifts can breach spend or capacity while unit cost remains stable.
• Use canary cohorts and explicit rollback thresholds for quality, latency, errors, saturation, spend, and telemetry health.
• Rebuild the frontier after model, prompt, retrieval, policy, tool, runtime, hardware, region, scheduler, cache, price, or traffic changes.
• Preserve raw-enough evidence and definitions to reproduce a decision without retaining sensitive content unnecessarily.