Regression Is a Projection

The hat matrix, orthogonal residuals, leverage, and why the least-squares line is not the principal axis.

Ch 3 3.5

Explorer

Mathematical Core

16 mindifficulty 3/5

Controls

True slopeβ₁1.200

True interceptβ₀2

Noiseσ2

Larger noise widens the gap between the OLS line and PC1.

Add one high-leverage point

A single observation far out in x.

Leverage point x16

Leverage point y4

Show vertical residuals

The distances OLS minimizes.

Overlay PC1

The line minimizing perpendicular distance.

Sample sizen60

Random seed

Every result on this page is a deterministic function of the seed and the controls.

The fit as a projection

ŷ = Hy is the orthogonal projection of y onto the column space of X.

OLS fitPC1 (total least squares)vertical residualshigh leverage

Residuals against fitted

Structureless if the model is right.

Leverage and influence

hᵢᵢ against Cook's distance.

β̂₀

2.175

true 2

β̂₁

1.120

true 1.20

PC1 slope

1.417

perpendicular-distance line

R²

0.698

r = 0.835

σ̂

2.186

true 2

tr(H)

equals p = 2 exactly

Σeᵢ

-1.55e-13

orthogonal to the intercept column

Σxᵢeᵢ

-3.77e-13

orthogonal to the x column

Why both dot products are zero

The normal equations are exactly the statement Xᵗe = 0. Every column of X is orthogonal to the residual vector, which is why the residuals sum to zero whenever an intercept is included — that is a consequence of the geometry, not an assumption.

Predict before you look

You fit y on x by OLS, then fit x on y by OLS and invert the second line. How do the two lines compare?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A simulated regression whose noise, geometry, and one optional high-leverage observation you control. The vertical residuals OLS minimizes are drawn explicitly, PC1 is overlaid for contrast, and the numerical readouts confirm the orthogonality conditions and tr(H) = p.

Projection onto the column space

Least squares chooses β̂ so that Xβ̂ is the point of the column space col(X) closest to y in Euclidean distance. That closest point is the orthogonal projection, so the residual e = y − Xβ̂ is perpendicular to every column of X: Xᵗe = 0. Those are the normal equations.

The projection matrix is H = X(XᵗX)⁻¹Xᵗ. It is symmetric and idempotent, its diagonal entries hᵢᵢ are the leverages, and its trace equals the rank p — the degrees of freedom the fit consumes. Cook's distance combines a residual with its leverage to measure how much removing an observation would move the whole fit.

OLS minimizes vertical distances because it treats x as fixed and known. PC1 minimizes perpendicular distances, treating both coordinates as noisy. With measurement error in x, OLS attenuates the slope toward zero while total least squares does not — the two lines coincide only in the noiseless limit.

β̂ = argmin ‖y − Xβ‖²  ⟺  Xᵗ(y − Xβ̂) = 0

H = X(XᵗX)⁻¹Xᵗ,  H² = H = Hᵗ,  tr(H) = p

hᵢᵢ = xᵢᵗ(XᵗX)⁻¹xᵢ,  Dᵢ = eᵢ² hᵢᵢ / (p σ̂² (1 − hᵢᵢ)²)

Try this

Read Σeᵢ and Σxᵢeᵢ: both are zero to machine precision at every setting. Explain why using Xᵗe = 0.
Raise the noise to 8 and compare β̂₁ with the PC1 slope. Which is closer to the true slope, and why?
Add the leverage point at x = 16, y = 4 and watch β̂₁ move; then move it to x = 0.5 and watch influence vanish.
Check that tr(H) stays at 2 while n changes from 15 to 400.
Set the true slope to 0 with large noise: R² is near zero although the model is exactly correct.

Common misconception

“The regression line and the first principal component are two names for the line of best fit.”

They minimize different distances. OLS minimizes the squared vertical gaps, so it depends on which variable you call the response — swapping x and y gives a different line. PC1 minimizes perpendicular gaps and is symmetric in the two variables. The gap between them grows with the noise-to-signal ratio, and with large noise OLS visibly under-estimates the slope while PC1 does not.

DSPA3 Chapter 3 — Matrix computing and regression