Where Leakage Hides

Impute before splitting, select features on all the data, or split a time series at random — and watch validation lie.

Ch 5 5.8

Misconception

Supervised Learning

22 mindifficulty 4/5

Controls

Dataset

Subjectsn60

Repeated measures per subjectr3

More repeats per subject means more to leak if folds ignore grouping.

Noise featuresp40

True signal strengthδ0

0 = no real relationship exists at all; the honest score should sit at chance.

Subject fingerprint strength0.900

Per-subject random offset shared by all of that subject's rows.

Batch/time drift strength0.500

Per-block random offset shared by all rows collected in that block.

Time blocks4

Pipeline / leakage toggles

Select features on all data

On = leaky: features are screened using rows that include the test fold.

Standardize on all data

On = leaky: mean/sd for scaling come from rows including the test fold.

Grouped CV (by subject)

On = honest: a subject's repeated rows all stay in one fold. Ignored when forward-chaining is on.

Forward-chaining split

On = honest: train only on the past, test only on the future block.

Estimation

Selected featuresk8

CV folds5

Repeated simulations60

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Reported CV vs. honest held-out score

Reported = k-fold CV on the training pool, with whatever leakage the toggles allow. Honest = the same fitted pipeline scored on freshly simulated, unrelated subjects.

Reported accuracy

58.3%

Honest accuracy

48.3%

Reported AUC

0.663

Honest AUC

0.452

left bar: reported (CV)right bar: honest (fresh data)

Optimism climbs with the number of noise features

Averaged over 6 quick simulations per point, current toggles held fixed. Watch what happens when selection or scaling happens before the split.

reported (CV)honest (fresh data)

Distribution of the gap across repeated simulations

Reported − honest accuracy, over 60 independent simulations from fresh seeds.

Mean reported

67.5%

Mean honest

49.9%

Mean gap

17.6%

Predict before you look

There is no real signal in the data (δ = 0), but you screen 300 noise features on the full dataset before a random 5-fold CV. What does reported CV accuracy look like?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A simulated dataset with mostly noise features, repeated measurements per subject, and a time index. Four common shortcuts — selecting features on all the data, scaling on all the data, random k-fold on grouped data, and random splits of a time series — are each toggled independently. For every configuration, the lab reports the optimistic cross-validated score next to an honest score measured on freshly simulated, unrelated subjects. The honest score is the truth; the reported score is only sometimes it.

Why the gap appears

Cross-validation is only an unbiased estimate of generalization error if every step that 'looks at the labels' — feature screening, scaling, imputation, hyperparameter choice — is refit inside each training fold and never touches the held-out fold.

When p candidate features are screened against n labels using all the data, some features correlate with the label purely by chance; the number of such flukes grows with p, so cross-validation on the selected features overstates accuracy more and more as p grows, even with zero true signal.

When observations are not independent — repeated measurements on the same subject, or autocorrelated time points — a random split can place near-duplicates of the same subject or the same time block on both sides. The model then partly memorizes a subject or era fingerprint rather than the general relationship, and only fails once it meets a genuinely new subject or a genuinely new time period.

\hat{\text{Acc}}_{CV} = \frac{1}{n}\sum_i \mathbb{1}[\hat{y}_{-f(i)}(x_i) = y_i]

\text{honest error} = \mathbb{E}_{(x,y)\sim \text{future}}\big[\ell(\hat f(x), y)\big] \neq \hat{\text{Acc}}_{CV}\ \text{when the fold boundary is crossed}

Try this

Set true signal strength to 0 and turn on only 'select features on all data'. Slide p up from 10 to 300 and watch reported accuracy climb while honest accuracy sits at chance.
With repeated measures per subject at 5 and subject fingerprint strength high, compare random k-fold to grouped CV. The reported number barely moves for grouped CV as you add noise — it can't cheat by recognizing subjects.
Turn on batch/time drift and compare a random split to forward-chaining. A model that looks great on a random split can be worthless once it has to predict a block it has never seen.
Now set true signal strength above 0 and repeat: honest accuracy should rise above chance and toggling the leakage switches should stop mattering nearly as much, because there is something real to find.

Common misconception

“As long as I evaluate on a held-out test set — or use k-fold cross-validation — my accuracy number is trustworthy.”

A held-out set only protects you if nothing about it influenced the model before that final evaluation. Feature selection, scaling, or imputation fit on the full dataset, or a split that fails to respect subjects or time, all let the test set quietly inform training. The reported number is then measuring memorization, not generalization.

Chapter 5.8, Data Leakage in Classification