LASSO Paths, Cross-Validation, and False Discoveries

Trace coefficients from saturated to empty, then discover that the cross-validated λ is tuned for prediction — not for finding the right variables.

Ch 11 11.3

Explorer

Validation & Feature Selection

25 mindifficulty 4/5

Controls

Regularization

log λ-2

λ = 0.1353 — larger λ means fewer surviving predictors.

Data-generating process

Observationsn60

Predictorsp30

Try p > n — OLS has no unique solution there, the LASSO still does.

Truly nonzero coefficients4

Noiseσ1

Predictor correlationρ0.400

Correlated predictors make selection unstable.

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Coefficient paths

Each line is one predictor's coefficient as λ varies. Coloured lines are truly nonzero.

True signalTrue zeroCV-optimal λ

Cross-validation curve

5-fold mean squared prediction error.

Estimated versus true coefficients

At the current λ.

TrueLASSO estimate

0.1353

CV optimum 0.1848

Selected predictors

truly nonzero: 4

True positives

signals recovered

False positives

noise variables selected

Max |shrinkage| on signals

0.573

the LASSO biases nonzero coefficients toward zero

OLS available?

yes, R² 0.921

unique solution exists

Predict before you look

Four of thirty predictors truly matter. You fit the LASSO at the cross-validated λ. How many of the twenty-six noise variables typically receive nonzero coefficients?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A sparse linear model you specify, fit by coordinate descent over a grid of penalties. Because the true support is known, the lab can score the selection directly: true positives, false positives, and the shrinkage applied to genuine signals.

The ℓ₁ penalty and what it costs

The LASSO minimizes squared error plus λ times the ℓ₁ norm of the coefficients. The ℓ₁ ball has corners on the coordinate axes, so the optimum frequently lands exactly on a corner and sets coefficients precisely to zero — this is why it selects while ridge only shrinks.

Coordinate descent solves it by cycling through coordinates and applying the soft-threshold operator, which subtracts λ from the magnitude of each partial correlation and clips at zero.

Cross-validation chooses λ to minimize predicted squared error. That objective is not support recovery: the minimizing λ is typically smaller than the one that would control false discoveries, so a handful of noise variables ride along. Meanwhile the same penalty that zeroes noise also biases the surviving coefficients toward zero.

β̂ = argmin ½‖y − Xβ‖² + λ‖β‖₁

soft(z, λ) = sign(z)·max(|z| − λ, 0)

λ → 0: OLS (when p < n);  λ ≥ λ_max: β̂ = 0

Try this

Slide log λ from −6 to 1 and watch the paths collapse one by one onto zero.
Move λ to the CV optimum and read the false-positive count — it is rarely zero.
Set p = 50 with n = 30 and confirm the OLS readout reports no unique solution while the path is unaffected.
Compare the estimated signal coefficients with the true values: they are systematically nearer zero.
Push ρ to 0.9 and redraw several times to see selection instability among correlated predictors.

Common misconception

“The variables selected at the cross-validated λ are the true predictors.”

Cross-validation optimizes out-of-sample prediction error, and prediction tolerates extra weakly correlated variables far more happily than it tolerates dropping a real one. So the CV-optimal model habitually includes false discoveries, and it also under-reports the size of true effects because of shrinkage. For inferential claims about which variables matter, use methods designed for that target — knockoffs, stability selection, or debiased inference — rather than reading off the LASSO's support.

DSPA3 Chapter 11 — Variable importance and controlled feature selection