Linear Models as Debugging Instruments

A baseline should be easy to fit, reproduce, and falsify. Start with an intercept-only predictor that encodes no feature signal. Then add a small, named feature set under a fixed split and loss. The comparison asks whether the data contains stable additive signal. It does not decide that linearity is true, causal, fair, or sufficient for the product.

1. 01Confirm the interceptReproduce the train mean or prevalence and its evaluation loss. If this fails, the target, weights, metric, or split plumbing is already inconsistent.
2. 02Fit one feature at a timeLook for implausibly strong proxies, reversed direction, unit mistakes, and features that work only in one time or source slice.
3. 03Fit the named small setTrack rank, conditioning, coefficient changes, and evaluation improvement under an explicitly versioned solver and penalty.
4. 04Inspect residual structureBucket residuals by prediction, time, target magnitude, entity group, and protected slices; random-looking aggregate residuals can hide a coherent subgroup failure.
5. 05Challenge with a richer modelRequire incremental value on the same causally valid evidence, then diagnose where it improves rather than accepting a single aggregate delta.

Residuals are observations about where a mapping misses, not automatic proof of why. A curved residual pattern can suggest a missing transform; time drift can suggest process change; a few influential points can suggest data errors or a legitimate rare regime. Each pattern proposes a next experiment that keeps target and split definitions fixed.

The reference artifact implements a deliberately small standard-library regression path. Its frozen contract binds target identity and unit, named features, dataset and feature snapshots, split IDs, required evaluation slices, solver convention, singular-pivot tolerance, ridge penalty, workload limits, aggregate and slice acceptance limits, and owners. It rejects mutable or duck-typed evidence, duplicate identities, non-finite numbers, wrong shapes and provenance, singular systems, arithmetic overflow, thin slices, and unbounded workloads. The entire dataset—including rows—receives a content identity that the report preserves.

1def format_example() -> str:
2    result = diagnose_linear_baseline(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_DATASET)
3    coefficients = ",".join(f"{value:.3f}" for value in result.coefficients)
4    return "\n".join(
5        (
6            "example=illustrative_only",
7            f"contract_version={ILLUSTRATIVE_CONTRACT.contract_version}",
8            f"dataset={result.dataset_id}",
9            f"evidence_id={result.evidence_id}",
10            f"coefficients=intercept,{ILLUSTRATIVE_CONTRACT.feature_names[0]}:{coefficients}",
11            f"train_rmse={result.train_rmse:.3f}",
12            f"evaluation_rmse={result.evaluation_rmse:.3f}",
13            f"mean_baseline_rmse={result.mean_baseline_rmse:.3f}",
14            f"evaluation_improvement={result.evaluation_improvement:.3f}",
15            f"lowest_improvement_slice={result.lowest_improvement_slice_id}:{result.lowest_slice_improvement:.3f}",
16            f"highest_rmse_slice={result.highest_rmse_slice_id}:{result.highest_slice_rmse:.3f}",
17            f"decision={result.decision}",
18        )
19    )

example=illustrative_only
contract_version=linear-diagnostic-v1
dataset=linear-baseline-run-0042
evidence_id=linear-diagnostic-evidence-v1@sha256:a88115a8fc3d8964e1994b25755cbba991de78ad0d6bfb6741eb9eeec44971fd
coefficients=intercept,queue-age-hours:1.000,2.000
train_rmse=0.000
evaluation_rmse=0.000
mean_baseline_rmse=6.083
evaluation_improvement=1.000
lowest_improvement_slice=established:1.000
highest_rmse_slice=new-account:0.000
decision=BASELINE_ESTABLISHED

1. 01Remove each suspicious featureA dramatic collapse after removing one identifier-like field is evidence to inspect lineage, not a reason to keep the shortcut.
2. 02Shuffle the target inside the training boundaryPerformance should collapse toward the comparator. Persistent signal suggests leakage through preprocessing, duplicates, or evaluation logic.
3. 03Perturb units and encodingConfirm predictions transform as expected and coefficient narratives do not survive a semantic unit change without revision.
4. 04Replay later and unseen groupsCompare residual distribution and slice error without changing the fitted model; this separates fit from transfer evidence.
5. 05Force arithmetic extremesReject NaN, infinity, booleans, overflow, and unsupported workload sizes instead of emitting plausible coefficients.

• Re-run the intercept and linear baselines on every target, feature, split, and data-pipeline revision before training expensive candidates.
• Store coefficients with units, encoding, centering, solver, penalty, split digest, and uncertainty; never publish a bare coefficient leaderboard.
• Compare complex models with the baseline on identical content-addressed evidence and required slices, including latency and operating cost.
• Monitor residual distribution, comparator improvement, slice gaps, conditioning, missingness, and feature range drift after launch.
• Treat a sudden baseline improvement as a possible leakage alarm and a sudden collapse as a possible data or target incident.