Holdout, k-Fold, LOO, Bootstrap: Estimating an Estimator

The same data and the same model give very different accuracy numbers — see each scheme's sampling distribution against the truth.

Ch 9 9.2

Comparator

Validation & Feature Selection

20 mindifficulty 4/5

Controls

Resampling scheme

Sample sizen40

Small n is where the schemes disagree most.

Polynomial degreed3

The truth is quadratic; degree 1 underfits, 9 overfits.

Noiseσ1.500

Irreducible MSE = σ² = 2.25

Replicate studies120

Random seed

Every result on this page is a deterministic function of the seed and the controls.

One study

A degree-d polynomial fitted to n points from a quadratic truth.

truthdegree 3 fit

Sampling distribution: 70/30 holdout

The estimate you would report, across 120 independent studies.

All schemes compared

Mean estimate ± 1 SD across studies.

2.70 ± 1.27 — 70/30 holdout
2.65 ± 0.72 — 5-fold CV
2.58 ± 0.70 — 10-fold CV
2.55 ± 0.63 — Leave-one-out
2.60 ± 0.69 — Bootstrap .632

True prediction MSE

2.522

4000 fresh points

Estimate mean

2.699

bias 0.177

Estimate SD

1.282

study-to-study variability

Apparent (training) MSE

2.414

always optimistic

Irreducible σ²

2.250

no model can go below

Root mean squared error of the estimator

1.294

bias and variance combined

Judge an estimator, not a number

Every scheme returns a number close enough to look authoritative. The question is its sampling distribution: holdout is nearly unbiased but wide, leave-one-out is narrow but correlated across folds, and the .632 bootstrap trades a little bias for the smallest spread. Choose on total error, not on bias alone.

Predict before you look

You have 40 observations. You fit a model and evaluate it with a single 70/30 split. Repeating the whole study on fresh data from the same population, how much would the reported MSE move?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A quadratic truth, a polynomial model of your chosen degree, and five resampling schemes. Because the data generator is known, the true prediction error can be computed on 4000 fresh points and each scheme's estimate can be judged for bias and for variability across hundreds of replicate studies.

What a validation estimate is estimating

Prediction error Err = E[(Y − f̂(X))²] over fresh draws decomposes into squared bias, variance of the fit, and the irreducible σ². The training (apparent) error underestimates Err by an optimism that grows with model complexity and shrinks with n.

A single holdout fits on a fraction of the data and evaluates on the rest. It is close to unbiased for a model trained on that smaller size, but its estimate rests on few test points, so its variance is large. k-fold reuses every observation once as a test case, averaging k estimates and cutting variance; the price is a mild pessimistic bias because each fit uses only (k−1)/k of the data.

Leave-one-out takes k = n: almost no bias from training size, but the n fits are nearly identical, so their errors are highly correlated and the variance reduction saturates. The .632 bootstrap blends the apparent error with out-of-bag error, because a bootstrap sample contains about 63.2% of the distinct observations.

Err = Bias² + Var + σ²

E[Err_train] = Err − optimism,   optimism ↑ with complexity

Err_.632 = 0.368 · Err_train + 0.632 · Err_oob

P(observation in a bootstrap sample) = 1 − (1 − 1/n)ⁿ → 1 − e⁻¹ ≈ 0.632

Try this

With n = 40 and holdout selected, read the estimate SD. How different could two honest reports be?
Switch through 5-fold, 10-fold, and leave-one-out and record mean and SD each time.
Compare the apparent MSE with the truth at degree 1, 3, and 9 — where is optimism largest?
Raise n to 200 and watch all five schemes converge.
Set degree 9 with n = 20: which scheme still gives a usable answer?

Common misconception

“I split the data 70/30, so the number I got is the model's accuracy.”

A single split gives one draw from a sampling distribution that can be very wide when n is small. In this lab, at n = 40 the holdout estimate routinely lands 30–50% above or below the true prediction error, so two competent analysts with identical data and identical models report different accuracies. k-fold averages over k splits and materially reduces that variability, and the estimate should always be reported with its uncertainty rather than as a point value.

DSPA3 Chapter 9 — Model assessment and validation