Systems labIntermediate100 min estimateOriginal publication

Calibration, Thresholds, and Decision Cost

A score orders examples; a calibrated probability supports expectation; a threshold applies a cost policy. Treating those as one number is how apparently accurate models make expensive decisions.

Authorship: InterviewsVector
Published / updated: 2026-08-20 / 2026-08-20
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Choose an action by comparing expected costs under a calibrated conditional probability, not by treating 0.5 as a universal boundary. For binary actions with false-positive cost CFP and false-negative cost CFN, the equal-cost threshold is CFP/(CFP+CFN), subject to a declared bounded probability-domain tie envelope. That calculation is admissible only when calibration evidence matches the event, population, horizon, cohort, model, calibrator, data, labels, evaluation window, and score region used by the decision. Bind the cost table, binning and metric conventions, owners, qualification thresholds, cost ratio, numeric resolution, and tie tolerance; require support in every contracted calibration bin; content-address the full serving score; and hold decisions when the calibration report is stale or unqualified.

Why this matters

The same model score can rationally trigger different actions when review capacity, missed-event harm, or population changes. Conversely, an impressive ROC area can coexist with probabilities that badly misstate expected frequency. Production systems need a visible boundary between score generation, probability qualification, and policy choice so a new cost table does not masquerade as a model improvement.

You will be able to

Distinguish ranking discrimination, probability calibration, and action thresholding.
Derive a binary expected-cost threshold from an explicit cost matrix.
Compute and interpret a stated binary Brier convention and reliability-bin gaps.
Bind calibration evidence, score applicability, costs, owners, versions, bin support, and tie behavior into one auditable policy.
Design recalibration and threshold changes that preserve causal and operational evidence.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Separate raw score, calibrated event probability, cost table, and action into versioned contracts.
02
Derive
Derive probability-quality metrics and the binary expected-cost equality threshold.
03
Build
Qualify held-out calibration evidence before applying a content-addressed serving score.
04
Stress
Attack empty bins, stale cohorts, reused score IDs, changed or subnormal costs, boundary ties, sub-resolution probabilities, and non-finite evidence.
05
Operate
Monitor calibration and realized decision cost by cohort, action, probability region, and label maturity.
06
Defend
Defend why the chosen action is lower expected cost and where that claim stops.

Separate three contracts that share one interface

The layers may ship together, but each has a different owner, version, and validation question.
Layer	Claim	Failure when confused
score	higher or lower reflects learned ordering	arbitrary margin is read as a probability
calibrator	probability groups match event frequency on a scope	old reliability mapping survives population change
decision policy	an action minimizes declared expected cost	0.5 or a capacity cutoff is presented as model truth

Calibration is conditional. A value of 0.7 is meaningful only with an event definition, population, time horizon, and observation process. If the label means chargeback within 30 days, a score evaluated against any dispute ever is not an alternate view of the same probability; it is another target.

Qualify probability evidence on its support

Brier = (1/N) Σᵢ (pᵢ − yᵢ)²

This lesson uses the mean binary squared error convention with y in {0,1}. Naming the convention avoids ambiguity with multiclass or differently scaled formulations.

gap(B) = | mean(pᵢ : i∈B) − mean(yᵢ : i∈B) |

A reliability bin compares average forecast with empirical frequency. Bin boundaries, support, cohort, and window are part of the claim; the artifact requires minimum evidence in every contracted bin.

Evidence check	Why it is necessary	Fail-closed response
held-out labels with full horizon	training targets and immature outcomes bias calibration	wait or use a mature window
support in the serving score bin	global averages do not justify an empty region	hold the action
same model and calibrator	a remapped score changes probability semantics	requalify
same cohort and event	base rate and conditional behavior move	split evidence or recalibrate
unique observations	duplicates manufacture confidence	reject the report

Derive the action boundary from expected cost

Move cost, not just the cutoff

Change a calibrated event probability and asymmetric false-positive and false-negative costs; predict the action at the cost-equality boundary, inspect both expected costs, and complete a graded policy check.

Turn a calibrated score into a costed action

Move the decision threshold and declare the cost of each error. The fixed evaluation cohort stays unchanged, so every result is a deterministic policy comparison rather than a new model run.

Act when score is at least0.5

False-positive costFalse-negative cost

Costs are dimensionless exercise inputs. They must represent the same decision scope and horizon before their sum is meaningful. The 900-case illustrative cohort contains empirical outcomes in nine score buckets.

Actions triggered: 500 / 900
Minimum tested threshold: 0.2
Avoidable cost: 4,680

Confusion evidence and decision costs at the current threshold
Outcome	Count	Unit cost	Cohort cost
True positive	350	0	0
False positive	150	12	1,800
False negative	100	80	8,000
True negative	300	0	0
Total decision cost			9,800

Cost comparison at the current threshold

The table prices errors at the selected boundary. Compare its total with the minimum tested cohort cost of 5,120 across eleven boundaries. Calibration supplies score meaning; the cost table supplies the action.

Is the current threshold minimum-cost for this declared cost table?

Yes, it reaches the minimumNo, another tested threshold costs less

Make a prediction, then check it against the current evidence.

cost(positive) = CFP(1−p); cost(negative) = CFN p

The simplified matrix assigns zero cost to correct actions. Real policies can require a fuller action-by-outcome table, capacity constraints, abstention, or downstream effects.

p* = CFP / (CFP + CFN)

At p*, the two idealized expected costs are equal. The artifact derives the boundary from scaled costs, rejects absolute costs, cost ratios, probabilities, and metric limits outside a contracted binary64 range, and caps the versioned probability-domain tie envelope at the declared numeric resolution so it cannot reach zero, one, or override a material cost difference. Outside that envelope it compares scaled expected costs so product underflow cannot invent a tie.

Threshold selection is a policy decision even when an optimizer searches the value. If a queue can review only 500 cases, the realized cutoff also depends on score distribution and capacity. Record whether the boundary came from an unconstrained cost model, a capacity allocation, a regulatory rule, or an experiment.

Build a policy that refuses unsupported scores

calibrated_threshold_policy.py

1    scale = max(contract.false_positive_cost, contract.false_negative_cost)
2    scaled_false_positive = contract.false_positive_cost / scale
3    scaled_false_negative = contract.false_negative_cost / scale
4    scaled_positive_cost = (1.0 - score.probability) * scaled_false_positive
5    scaled_negative_cost = score.probability * scaled_false_negative
6    if not math.isfinite(scaled_positive_cost) or not math.isfinite(
7        scaled_negative_cost
8    ):
9        raise OverflowError("scaled expected decision cost overflowed")
10    positive_cost = scaled_positive_cost * scale
11    negative_cost = scaled_negative_cost * scale
12    if not math.isfinite(positive_cost) or not math.isfinite(negative_cost):
13        raise OverflowError("expected decision cost overflowed")
14    if math.isclose(
15        score.probability,
16        contract.threshold,
17        rel_tol=0.0,
18        abs_tol=contract.tie_tolerance,
19    ):
20        action = contract.tie_action
21    elif scaled_positive_cost < scaled_negative_cost:
22        action = contract.positive_action
23    else:
24        action = contract.negative_action

Expected output

example=illustrative_only
contract_version=calibrated-threshold-v1
calibration_status=QUALIFIED
brier=0.125
max_gap=0.000
threshold=0.200
expected_costs=auto-clear:2.800,manual-review:1.300
action=manual-review

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/calibrated_threshold.

The excerpt is literal source. The preceding path requires concrete immutable observations; binds evaluation ID and window, event, population, horizon, cohort, model, calibrator, data, labels, binary64 resolution, and qualification limits; rejects booleans, non-finite values, sub-resolution nonzero probabilities, subnormal costs, and zero or sub-resolution metric gates; enforces unique observations and entities; preserves every bin count and gap in a content-addressed report; validates report identity; and checks that the serving score's bin has contracted support. The decision retains a content ID for the full score evidence, so reusing a human-readable score ID cannot make two probabilities look identical.

Attack applicability and decision identity

01Leave one bin emptyQualification must fail rather than producing a global report that later authorizes an unsupported serving score.
02Reuse the score IDCreate two different probabilities with the same display ID and prove the decision binds different full evidence content IDs.
03Change only the cost tableInvalidate the old report-policy pairing and recompute the threshold under the approved version.
04Probe the bounded tie envelopeTest zero, one, the stored threshold, and values just outside the versioned tolerance; prove endpoints cannot tie and reject subnormal costs, unrepresentable ratios, and sub-resolution probabilities.
05Forge stale scopeChange one event, cohort, evaluation ID, label, model, calibrator, or data version and reject before calculating cost.

Operate calibration and policy as separate releases

Change	Required evidence	Likely owner
model score distribution	discrimination plus fresh calibration support	model owner
calibrator	held-out reliability and score-region coverage	measurement owner
cost table	approved impact model and sensitivity analysis	policy and finance owners
threshold or capacity	action-volume, queue, and outcome experiment	operations owner
event or horizon	new target and mature labels	label owner

Monitor counts, event rates, Brier score, bin gaps, score coverage, action volume, realized costs, abstentions, and overrides by cohort. Delay outcome claims until labels mature. If a policy changes who receives review or treatment, later labels may become selectively observed; carry that issue into the drift and feedback contract instead of recalibrating on the convenient subset.

Operate at three altitudes

Production lens

— Log raw-score, calibrator, cost-table, threshold-policy, event, cohort, label, and action identities with every decision.
— Retain per-bin support and reliability; fail closed when a serving probability falls outside qualified evidence.
— Monitor action volume, queue saturation, overrides, mature outcomes, and realized costs alongside probability metrics.
— Separate model, calibrator, and policy rollouts so regressions and ownership remain attributable.

Staff lens

— Assign explicit owners for event semantics, labels, calibration, costs, threshold authority, operations capacity, and appeals.
— Require cost sensitivity analysis because estimated harms are uncertain and can differ across cohorts.
— Fund label-maturity and selective-observation infrastructure before demanding real-time probability guarantees.
— Treat a threshold override as a versioned policy event with an expiry and measured consequence.

Interview defense

A classifier has strong AUC, but the business asks what threshold to use. How do you answer?

I would not choose 0.5 from AUC. I would define the event, population, horizon, and action costs; verify held-out probability calibration for the exact model and cohort with mature labels and support in the relevant score regions; then compare expected costs or solve the declared constrained allocation. I would bind the cost table, binning and metric conventions, numeric resolution, threshold boundary, bounded tie envelope, owners, versions, and score evidence. Finally I would validate action volume, capacity, slice impact, and realized outcomes in a controlled rollout because a lower offline expected cost does not prove the policy's downstream effect.

Expect the interviewer to press on

— Why can a monotonic score transform preserve AUC but break a cost decision?
— What do you do when the serving score lies in an empty calibration bin?
— How does limited review capacity change the threshold problem?

Misconceptions to remove

“A model score of 0.8 means an 80% event chance.”

Only a probability estimator qualified on the same event, population, horizon, and score region supports that interpretation.

“The best threshold is 0.5.”

The boundary depends on calibrated probabilities, action costs, constraints, and a declared tie rule. Equal costs yield 0.5 only in a simplified binary matrix.

“Recalibration is cosmetic because ranking is unchanged.”

Actions driven by probability or expected cost can change even when rank order is identical.

Check your model

1. With false-positive cost 2 and false-negative cost 8, what is the simplified equal-cost threshold?

2/(2+8) = 0.2. At the stored boundary and inside its declared tolerance envelope, the tie action applies; operational constraints can require a different policy.

2. Why preserve per-bin support in the calibration report?

A global Brier score or maximum gap cannot justify a probability region with no evidence. The serving decision must be within the report's applicability support.

3. Why content-address the complete score rather than store only score_id?

A reused display ID can refer to different probabilities or versions. The content identity binds the actual evidence that produced the action.

Prove the mechanism

Build a binary calibrated-threshold policy for an invented event. Bind held-out observations to event, population, horizon, cohort, evaluation, model, calibrator, data, label, numeric-resolution, cost-ratio, and tie-envelope versions; require support in every bin; preserve bin gaps and counts; derive the expected-cost threshold from a versioned cost table; and test stale reports, unsupported bins, reused score IDs, boundary-adjacent values, endpoints, subnormal costs and probabilities, zero metric gates, booleans, non-finite values, duplicates, and changed owners.

Add a production constraint

Add abstention and a capacity constraint. Define an action-by-outcome cost table, solve a deterministic allocation over content-addressed scores, expose sensitivity to uncertain costs, and design a shadow evaluation that measures mature outcomes without learning only from selected cases.

Artifact: Calibrated threshold policy

courses/ai-engineering/reference-impl/calibrated_threshold/calibrated_threshold_policy.py

Download reference implementation

Primary references and next links

References

1. Verification of Forecasts Expressed in Terms of Probability
Brier. Primary publication establishing the quadratic probability-verification score lineage used with an explicit binary mean convention here.
2. Predicting Good Probabilities with Supervised Learning
Niculescu-Mizil and Caruana. Primary empirical paper supporting the distinction between discrimination and calibrated probability estimates.
3. The Foundations of Cost-Sensitive Learning
Elkan. Primary paper supporting expected-cost reasoning under an explicit classification cost matrix.

Continue through the graph

Probability for Decisions, Not Decoration →
Review conditional probability and expected-cost foundations.
When the Product Needs Ranking, Not Classification →
Separate probability actions from ordered product utility.
Drift, Feedback Loops, and Delayed Labels →
Operate probability policies when labels and populations move.

Glossary: calibration · reliability diagram · Brier score · decision threshold · expected cost · false positive · false negative · applicability support