Failure labIntermediate105 min estimateOriginal publication

Split Data by Causality, Not Convenience

A split is an executable counterfactual: what could the learning system have known before each simulated decision?

Authorship: InterviewsVector
Published / updated: 2026-08-20 / 2026-08-20
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Choose a split by reconstructing the causal information boundary at deployment. Every feature must exist no later than its decision, every label must be evaluated only after its complete outcome horizon, training labels must mature before validation decisions, and validation labels must mature before the final test. Keep dependent entities and groups in one partition, bind source lineage and target scope, and content-address the complete manifest. Random assignment is valid only when exchangeability and independence actually describe the use case; it is never a default proof against leakage.

Why this matters

Leakage produces the most expensive kind of success: an offline result that looks unusually strong, wins organizational confidence, and cannot be reproduced online. Duplicate users, future aggregates, globally fitted preprocessing, early negative labels, and source-family overlap can all survive a clean row-level random split. A causal manifest makes each source of information answer when and for whom it was available.

You will be able to

Translate a deployment timeline into feature, decision, event, label-maturity, and evaluation-cutoff timestamps.
Select time, entity, and group isolation rules from the dependency structure rather than a preferred percentage.
Distinguish future-feature leakage, label censoring, preprocessing leakage, and dependency leakage.
Build a content-addressed manifest that fails closed on scope, provenance, overlap, and bounded workload violations.
Operate split policies as versioned data products with lineage, tests, and final-holdout governance.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Draw the deployment decision, upstream observations, outcome window, identities, groups, and source lineage.
02
Derive
Derive which facts exist at each decision and when each label becomes final.
03
Build
Create a versioned manifest with ordered windows, embargo, scope, group isolation, and content identity.
04
Stress
Inject future features, immature negatives, reused entities, crossing groups, source swaps, and cutoff violations.
05
Operate
Generate splits from canonical timestamps, preserve manifests, and guard the final test from adaptive reuse.
06
Defend
Explain why every row was admissible and which deployment counterfactual the score estimates.

Model the information graph

For each example, draw the entity and group, the feature events it depends on, the simulated decision time, the possible target event, and the time at which the label is final. Add any preprocessing fit, aggregate, lookup table, teacher prediction, or human review that can transmit information between rows. The split must cut this graph so the simulated learner receives no path from its future or evaluation outcomes.

feature_as_of(i) ≤ decision_time(i)

A feature is admissible only if the production system could compute the same value by the decision. A source column timestamp is insufficient when its aggregate includes later records.

label_as_of(i) ≥ decision_time(i) + horizon

For a full-horizon binary event label, absence becomes a mature negative only after the entire horizon. A positive also records its event time within that horizon so outcome semantics are unambiguous.

Identity	Dependency it captures	Isolation question
record	one prediction opportunity	is it duplicated or derived twice?
entity	persistent subject state	will production see known or unseen subjects?
group	shared household, site, vendor, device, or author	can related entities transmit signal?
time	market, policy, season, and availability	must the model predict later periods?
provenance	shared source and transformation lineage	was evaluation information used upstream?

Derive the deployment simulation

A split can combine boundaries; its purpose is to imitate the actual unknown, not satisfy a universal recipe.
Deployment question	Primary boundary	Additional control
later decisions for new entities	time then entity	group isolation and label maturity
later decisions for known entities	time	fit entity-history features only from the past
new hospitals, stores, or tenants	group	hold out whole operational domains
same-period exchangeable units	entity-aware random	prove no shared source or preprocessing fit
policy launch after historical training	forward time	embargo labels and freeze test

An embargo is not decorative empty space. It ensures an earlier partition's full-horizon labels were available before the next partition's simulated decisions. If a 30-day target is evaluated on a training decision from day 60, that negative cannot be used to select a model at day 70. It matures at day 90, and any operational delay belongs in the embargo too.

max(label_as_of_train) + embargo ≤ min(decision_validation)

This version of the policy demands completed training labels before validation begins. Apply the corresponding rule between validation and test, then bind a final test evaluation cutoff.

Move the boundary and predict the leak

Causal split laboratory

Move identity, group, feature-time, and label-maturity boundaries to predict which evaluation rows remain causally admissible.

Make the split answer the deployment question

Choose what the model must generalize to, then inspect whether the split blocks the causal paths that would carry identity or future evidence into training.

Deployment target

Later events from this serviceEvaluation must not send future information into training.First event from an unseen accountEvaluation identities must be absent from training.Future events from unseen accountsBoth time direction and identity isolation must hold.

Candidate split

Random rowsShuffles observations without respecting identity or event time.Group by accountKeeps each account in one partition, but mixes event time.Chronological cutoffTrains only on earlier events, but allows repeat accounts.Group, then apply a time cutoffHolds out accounts and keeps every evaluation event later.

Evaluation rows from repeat identities65%

This cohort contains 80 evaluation rows. The repeat-identity share changes the leakage exposed by row or chronological splits; it does not change which causal rule the deployment target needs.

Identity overlap: 52 / 80; Evaluation identities already appear in training.
Reverse-time rows: 24 / 80; Some evaluation rows predate training rows.
Illustrative score: 84.3%; May include structural optimism from either violation.

Audit rule: A deployment claim is supportable only when every identity and time boundary required by the selected target is closed. Use the counts above to decide whether this split meets that rule.

Can this evaluation support the selected deployment claim?

Yes, the split supports the claimNo, a required causal boundary is open

Make a prediction, then check it against the current evidence.

Build a causal split validator

The reference artifact implements one explicit convention: features are available no later than the decision; all binary labels are observed through the full horizon; entities and groups are exclusive; train, validation, and test use bounded forward windows; and earlier labels plus an embargo precede the next decision window. The content-derived contract ID binds task, target, population, rules, boundaries, cutoff, data revision, provenance, and owners. Each row repeats the critical scope fields, while the complete immutable manifest receives its own evidence digest.

causal_split_validator.py

1def format_example() -> str:
2    audit = validate_causal_split(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_MANIFEST)
3    counts = ",".join(f"{name}:{count}" for name, count in audit.counts)
4    return "\n".join(
5        (
6            "example=illustrative_only",
7            f"contract_version={ILLUSTRATIVE_CONTRACT.contract_version}",
8            f"manifest={audit.manifest_id}",
9            f"evidence_id={audit.evidence_id}",
10            f"counts={counts}",
11            f"minimum_boundary_gap_days={audit.minimum_boundary_gap_days}",
12            f"group_overlap={audit.group_overlap_count}",
13            f"entity_overlap={audit.entity_overlap_count}",
14            f"decision={audit.decision}",
15        )
16    )

Expected output

example=illustrative_only
contract_version=causal-split-v1
manifest=renewal-causal-manifest-0042
evidence_id=causal-split-evidence-v1@sha256:d368e9207aa516c01bd026c380e39a028a9f4e14439ba79e7149eae9827eb1cf
counts=train:2,validation:2,test:2
minimum_boundary_gap_days=40
group_overlap=0
entity_overlap=0
decision=PASS

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/causal_split.

01Generate, do not hand-editDerive a manifest from canonical event time, feature lineage, and identity tables, then store it as a release artifact.
02Validate the rules you claimReject unknown rule names rather than recording a convention the implementation does not enforce.
03Digest the rowsMake record membership, timestamps, outcomes, and scope part of the manifest identity so a reused label cannot conceal changed evidence.

Stress distinct leakage mechanisms

Leak	Tempting implementation	Fail-closed test
future feature	current account state joined to old decisions	feature-as-of exceeds decision
censored label	today's non-event treated as a 30-day negative	label-as-of precedes horizon maturity
entity memorization	multiple sessions randomly divided	entity appears across partitions
group transmission	sibling devices or documents separated	group appears across partitions
preprocessing leakage	scaler or vocabulary fit globally	fit provenance includes evaluation rows
test adaptation	test metric guides features repeatedly	final evidence is no longer decision-independent

01Backfill one featureKeep its row timestamp unchanged but advance its true source-as-of time. The validator should use availability, not table location.
02Move a group, not a rowCopy a new entity from a training region into validation. Entity isolation passes; group isolation must fail.
03Advance the final test windowRequire the test evaluation cutoff to mature the last decision's entire target horizon before declaring the manifest evaluable.
04Swap source lineageReject rows from an unapproved dataset revision even when all timestamps and identities look plausible.

Operate splits as governed data products

Persist the contract, manifest digest, generation query or job revision, excluded-row reasons, and source lineage with every experiment.
Fit imputers, encoders, scalers, feature selection, calibration, and learned thresholds inside the training boundary; treat their fitted state as model state.
Use validation for choices and keep the final test inaccessible to routine tuning. A test seen repeatedly becomes another validation set.
Monitor entity and group overlap counts, time coverage, label maturity, partition sizes, outcome rates, and slice support on every regeneration.
When deployment serves several unknown types, maintain several evaluation views instead of forcing one split to answer incompatible questions.

Artifact	Identity to preserve	Rebuild trigger
split contract	scope, rules, boundaries, owners	deployment question changes
row manifest	content digest and lineage snapshot	data or eligibility changes
fitted preprocessing	training-manifest digest	training membership changes
validation results	model plus manifest digest	candidate or policy changes
final test result	decision-independent release evidence	declared release evaluation

Operate at three altitudes

Production lens

— Store feature event time separately from ingestion and materialization time, and validate the time that the actual value became available.
— Bind task, target, population, full-horizon label semantics, dataset revision, and provenance into both contract and manifest evidence.
— Generate entity- and group-exclusive partitions before fitting any learned preprocessing state.
— Protect the final test from iterative selection and retain every rejected-row reason for audit and repair.

Staff lens

— Define reusable identity and group taxonomies across data products; leakage policy cannot scale when every team invents a different dependency unit.
— Require data producers to expose event-time and availability-time lineage so consumers do not infer causality from warehouse timestamps.
— Maintain multiple evaluation views when products face both familiar-entity and unseen-domain deployment, and prevent one aggregate from erasing the distinction.

Interview defense

You are predicting whether an account will cancel within 30 days. How would you construct train, validation, and test data?

I would start from the production decision time and build features using only values available then. Each row would carry entity and dependency-group IDs, feature-as-of time, event time if positive, and label-as-of time. A negative is admissible only after 30 days. I would use forward windows, ensure training labels plus an operational embargo mature before validation decisions and validation labels mature before test decisions, isolate entities and shared groups according to the deployment unknown, fit preprocessing only on training, bind scope and lineage into a content-addressed manifest, and reserve the final test for release rather than iteration.

Expect the interviewer to press on

— When would an entity-aware random split be appropriate?
— Why can a time split still leak?
— How do delayed labels change the boundary?
— What belongs in the split manifest identity?

Misconceptions to remove

“Shuffling before splitting removes bias.”

Shuffling changes assignment, not dependency. Related entities, future information, censored labels, and globally fitted state can still cross the boundary.

“A feature is safe if its database column predates the label column.”

The value may have been recomputed from later events. Safety depends on value-level availability and lineage at the decision time.

“Once test rows are held out from training, test use is unlimited.”

Repeated decisions based on test results adapt development to that set. Preserve a final decision-independent evaluation or explicitly reclassify it as validation.

Check your model

1. Why must a 30-day non-event label have a label-as-of time at least 30 days after the decision?

Before the horizon closes, the event can still occur. Counting the row as negative censors late events and biases the evaluation.

2. A customer never crosses partitions, but two customers from the same household do. What assumption failed?

Entity isolation was too narrow. The household is a dependency group that can transmit shared signal, so it must be isolated when deployment requires unseen households.

3. Why should fitted preprocessing carry the training-manifest digest?

Its statistics, vocabulary, or selected features were learned from particular rows. The digest proves that evaluation data did not influence that fitted state.

Prove the mechanism

Extend the validator with a preprocessing manifest whose fit-row digest must equal the training partition digest. Add adversarial tests for a globally fitted scaler and a target encoder that reads validation labels.

Add a production constraint

Support a rolling-origin evaluation with several content-addressed windows while preserving full-horizon label maturity, group isolation, bounded workload, and a separate final release holdout.

Artifact: Causal split validator

courses/ai-engineering/reference-impl/causal_split/causal_split_validator.py

Download reference implementation

Primary references and next links

References

1. Leakage in Data Mining: Formulation, Detection, and Avoidance
Kaufman, Rosset, Perlich, and Stitelman. Primary ACM paper formalizing leakage around information illegitimately available to the learner.
2. A Critical Study on Data Leakage in Recommender System Offline Evaluation
Ji, Sun, Zhang, and Li. Primary paper demonstrating future-item leakage when offline evaluation ignores the global timeline.
3. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts et al.. Primary study of blocked validation for dependent observations and structured deployment questions.
4. TimeSeriesSplit and GroupKFold
scikit-learn. Official documentation for time-ordered and group-exclusive cross-validation strategies.

Continue through the graph

The Generalization Contract →
Anchor the split in the exact population, task, horizon, and loss the evaluation claims to represent.
Linear Models as Debugging Instruments →
Use a transparent baseline to probe the resulting partitions for shortcut signal and target failures.
Academy roadmap →
Place causal validation inside the full learning-systems arc.

Glossary: data leakage · feature availability · label maturity · embargo · entity isolation · group isolation · forward split · manifest