The Generalization Contract

Begin at the moment a prediction will be consumed. Identify the eligible entity, the decision being supported, the latest information legally and operationally available, the event and horizon, and the consequence being measured. Only then select a dataset. Starting from a convenient table encourages the table's historical accidents to masquerade as the deployment population.

A small train-to-test gap can coexist with high risk when both scores are poor. A low test risk can still fail deployment if the sampling frame excludes hard entities or the production loss weights errors differently. Conversely, a modest aggregate regression may conceal a severe protected-slice failure. Keep absolute risk, transfer assumptions, slice evidence, and uncertainty separate.

The reference artifact supports one explicit binary Brier-risk convention. Its frozen contract fixes the task, population, decision point, 30-day horizon, label definition, model, dataset, sampling frame, required slices, thresholds, and owners. Every row carries stable record and entity identities, its feature cutoff, event time when positive, full-horizon label-as-of time, prediction, outcome, and slice membership. The full evidence object receives a SHA-256 content identity, so two evaluations cannot share a run label while silently changing predictions or rows.

1def format_example() -> str:
2    audit = audit_generalization(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_EVIDENCE)
3    return "\n".join(
4        (
5            "example=illustrative_only",
6            f"contract_version={ILLUSTRATIVE_CONTRACT.contract_version}",
7            f"evaluation={audit.evaluation_id}",
8            f"evidence_id={audit.evidence_id}",
9            f"records={audit.record_count}",
10            f"empirical_risk={audit.empirical_risk:.3f}",
11            f"worst_slice={audit.worst_slice_id}:{audit.worst_slice_risk:.3f}",
12            f"decision={audit.decision}",
13        )
14    )

example=illustrative_only
contract_version=generalization-contract-v1
evaluation=renewal-evaluation-run-0042
evidence_id=evaluation-evidence-v1@sha256:30095ac28b8a270520044ad288c168de952df89cd2b21a0ef824fa01b551cd38
records=4
empirical_risk=0.075
worst_slice=low-volume:0.125
decision=PASS

1. 01Freeze the claimContent-address every assumption that changes what a score means, including slice requirements and acceptance limits.
2. 02Validate before aggregatingReject wrong populations, repeated entities, future features, immature labels, invalid numbers, undeclared slices, and stale provenance before allocating reports.
3. 03Preserve the evidence identityReturn the full content digest beside the friendly evaluation name so an operator can reproduce the exact decision input.

1. 01Censor a negative rowSet its label-as-of time before the full outcome horizon. A correct audit refuses to count absence of an event as a final zero.
2. 02Reuse one entityDuplicate a customer under a fresh record ID. The sample size increases, but independent evidence does not; reject the row under this contract.
3. 03Keep a run name, alter one predictionThe human label may remain unchanged, but the content identity must change and bind the new decision.
4. 04Hide a rare sliceRefuse an aggregate score when a contractually required slice lacks enough mature examples.

• Store the contract ID, evidence ID, model digest, dataset snapshot, feature view, label policy, and code revision with every gate decision.
• Report sample counts, time coverage, missingness, and risk intervals by required slice; a point estimate without support can invite false precision.
• Separate model acceptance thresholds from alert thresholds. One governs release authority; the other governs investigation after release.
• Re-evaluate when acquisition, eligibility, product policy, label adjudication, feature availability, or model selection changes—not only when weights change.
• Keep rollback tied to the model-plus-policy bundle whose evidence passed, rather than to a floating model name.