Overparameterized and Still Improving

Push past the interpolation threshold and watch test error fall a second time.

Ch 14 14.8

Misconception

Deep Learning

22 mindifficulty 5/5

Controls

Training pointsn25

Noise levelσ0.300

Max featuresp_max200

Sweep runs from p=2 up to this many random features.

Ridge penaltyλ0

λ = 0 is ridgeless (minimum-norm interpolation).

Repetitions to average6

Feature type

Fitted-view feature countp25

interpolation threshold (p = n)

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Train and test risk vs. p / n

log₁₀(MSE). The dashed vertical line marks the interpolation threshold p = n.

train risktest riskp = n thresholdirreducible noise floor

Minimum-norm coefficient size

||β|| spikes exactly where the fit first interpolates.

Fitted function at p = 25

interpolation threshold (p = n)

true f(x)fitted functiontraining points

Peak location

p/n = 0.08

p = 2

Test risk at p = n

0.4826

the interpolation spike

Test risk at p = p_max

0.3723

second descent

‖β‖ at p = n

164.624

vs. ‖β‖ at p_max = 339.297

Two descents, one curve

Test risk falls as p grows from far below n (classical regime), spikes to its worst value near p = n where the fit exactly interpolates n noisy points with the least possible slack, then falls a second time as p keeps growing past n — because among the infinitely many interpolating solutions, the minimum-norm one gets smoother as there is more room to spread the fit thinly.

Predict before you look

As the number of random features p grows past the number of training points n, test risk...

Scenarios

Teacher-ready starting points.

Documentation

What this shows

Random-feature regression on a fixed nonlinear target: n training points are fit with p random ReLU or Fourier features by minimum-norm (ridgeless or ridge) least squares, sweeping p from far below n to far above it. Train risk falls monotonically, but test risk spikes right at p = n — the interpolation threshold — and then descends a second time as p keeps growing.

Why interpolation is worst, not best

With p random features φ₁(x),…,φ_p(x), fitting Xβ ≈ y by minimum-norm least squares gives the dual solution β = Xᵀ(XXᵀ + λI)⁻¹y, valid for any p relative to n. At λ → 0 and p ≥ n this is the minimum-ℓ₂-norm interpolant: the unique zero-training-error solution with the smallest ‖β‖.

Near p = n, X is barely invertible in the relevant directions: the smallest singular values of the n×p design shrink toward zero, so ‖β‖ must grow enormously to force the residual to exactly zero. That coefficient blow-up is what drives the test-risk spike.

Past p = n there are many interpolating solutions, and the minimum-norm one increasingly resembles a smooth kernel regression as p grows — because spreading the fit across many small, well-conditioned directions requires less total coefficient mass than squeezing through a nearly-singular n×n bottleneck.

β̂ = Xᵀ(XXᵀ + λI)⁻¹y

λ → 0, p ≥ n:  β̂ = argmin{‖β‖ : Xβ = y}

peak ≈ where σ_min(X) → 0, i.e. p ≈ n

Try this

Load the classic-double-descent preset and watch the test-risk curve: falling, spiking at p/n = 1, falling again.
Turn ridge λ up from 0 to 0.05–0.1 and watch the spike flatten while the far-right test risk barely changes.
Drag the fitted-view slider from p ≪ n through p = n to p ≫ n and watch the curve go from too-simple, to wildly interpolating every noisy point, to smooth again.
Increase the noise level and compare how much the peak height changes versus how much its location (always p ≈ n) changes.
Switch feature type between ReLU and Fourier — the double-descent shape survives the change of basis.

Common misconception

“Test error is U-shaped in model capacity: more parameters than data always means overfitting and gets worse forever.”

Test error is worst exactly at the interpolation threshold p = n, not beyond it. Once the model has enough capacity to interpolate the training data in many different ways, the minimum-norm (or otherwise regularized) choice among those solutions gets smoother — not wilder — as capacity keeps growing, producing a second descent in test risk.

DSPA3, Chapter 14.8