Conditioning, Collinearity, and Unstable Coefficients

Watch the condition number of a design matrix explode and see exactly which quantities become unidentifiable — and which do not.

Ch 3 3.5

Diagnostic

Mathematical Core

20 mindifficulty 3/5

Controls

Predictor correlationρ0.950

Pushes the design matrix toward singularity.

Noise scaleσ1.500

Sample sizen120

RefitsR200

Independent datasets, each refit from scratch.

Solver

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Design geometry

Each point is one row of the design matrix.

As ρ → 1 the cloud collapses onto a line: the columns nearly coincide, so the data cannot distinguish β₁ from β₂.

Sampling distributions of the coefficients

200 independent datasets, refit each time. True β₁ = 1.5, β₂ = −1.

β̂₁ across refitsβ̂₂ across refits

Condition number κ(X)

σ_max / σ_min

VIF

10.26

1/(1−ρ²)

β̂₁ (this dataset)

1.716

true 1.5

β̂₂ (this dataset)

-1.181

true −1

SD of β̂₁ across refits

0.437

SD of β̂₁ + β̂₂

0.148

the sum stays stable

R²

0.205

unharmed by collinearity

Residual SE

1.573

true σ = 1.50

Predict before you look

Two predictors are correlated at ρ = 0.999. Across many refits, what happens to the average of β̂₁?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A three-column design matrix whose two predictors have a correlation you control. The lab refits the same model on many fresh datasets so you can see the sampling distribution of each coefficient, alongside the condition number of the design.

Why κ(X) governs the damage

Least squares projects y onto the column space of X. When two columns are nearly parallel, many very different coefficient vectors give almost the same fitted values, so the solution is poorly determined even though the projection itself is fine.

The condition number κ(X) = σ_max/σ_min quantifies this: a relative perturbation in the data can be amplified by κ in the solution. Forming XᵗX squares the condition number, which is why the normal equations lose roughly twice as many digits as a QR factorization.

Collinearity inflates variance without introducing bias — the average of β̂ over refits stays at the true value while its spread grows like 1/(1−ρ²).

β̂ = (XᵗX)⁻¹Xᵗy = R⁻¹Qᵗy

κ(XᵗX) = κ(X)²

Var(β̂ⱼ) = σ² / [(n−1)·Var(xⱼ)·(1−R²ⱼ)],  VIF = 1/(1−R²ⱼ)

Try this

Slide ρ from 0 to 0.99 and track the condition number and the width of the two coefficient histograms.
At ρ = 0.99, compare the SD of β̂₁ with the SD of β̂₁ + β̂₂.
Check that the coefficient histograms remain centred on the true values at every ρ.
Watch R² and the residual SE while ρ increases — prediction quality is essentially untouched.
Increase n at fixed ρ: variance falls like 1/n, but the VIF penalty is unchanged.

Common misconception

“Collinearity biases my regression coefficients, so I should drop one predictor.”

Least squares stays unbiased at any ρ < 1; only the variance inflates. Dropping a genuinely relevant predictor introduces real omitted-variable bias, trading a variance problem for a bias problem. If prediction is the goal, collinearity may be harmless; if a single coefficient must be interpreted, the fix is better data or a regularized/identifiable target such as the sum.

DSPA3 Chapter 3 — Linear algebra and regression