PCA Geometry: Projection, Scale, and What Variance Means

Rotate a data cloud, change a measurement unit, and see principal components move — the projection is geometry, not magic.

Ch 4 4.2

Explorer

Mathematical Core

18 mindifficulty 3/5

Controls

Cloud orientationθ30°

Major-axis spread4

Minor-axis spread1

Unit of x₁ (e.g. m → mm)8

A pure change of measurement unit — no new information.

Standardize columns

Divide each variable by its SD before decomposing.

Show projection residuals

Perpendicular drops onto PC1.

Pointsn260

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Principal axes

PC1 minimizes the sum of squared perpendicular distances — not vertical ones.

PC1PC2Projection residuals

Scree

Share of total variance per component.

PC1 scores

The one-dimensional summary you would carry forward.

PC1 variance share

99.8%

PC2 variance share

0.2%

PC1 direction

1, 0.07

loadings on (x₁, x₂)

PC1 angle

3.8°

cloud drawn at 30°

SD of x₁

24.81

raw units

SD of x₂

2.01

raw units

Predict before you look

You run PCA on the covariance matrix. Then you re-express one predictor in millimetres instead of metres and run it again. What happens to PC1?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A two-dimensional cloud you can rotate and stretch, with the principal axes recomputed live. The dashed segments are the perpendicular residuals that PC1 minimizes, which is what distinguishes PCA from least-squares regression.

Eigenvectors of the covariance matrix

Centre the data and form the covariance matrix S. Its eigenvectors are the principal directions and its eigenvalues are the variances along them, so the share of variance carried by component k is λₖ divided by the trace of S.

PC1 is the direction maximizing projected variance, which is equivalently the line minimizing summed squared perpendicular distances. Regression minimizes vertical distances instead, so the two lines differ whenever the noise is not negligible.

Covariance carries units. Multiplying x₁ by 1000 multiplies its variance by 10⁶, so PC1 will align with x₁ almost regardless of structure. Standardizing means decomposing the correlation matrix instead, which removes the unit dependence.

S = (1/(n−1)) XᶜᵗXᶜ,   S vₖ = λₖ vₖ

share(k) = λₖ / Σⱼ λⱼ

PC1 = argmax_{‖v‖=1} Var(Xᶜv) = argmin_{‖v‖=1} Σᵢ dist⊥(xᵢ, span v)²

Try this

Set the x₁ unit to 1 and note PC1's angle; slide it to 20 and watch PC1 swing toward the x₁ axis.
Now switch standardization on and repeat the same slide — the angle stops moving.
Make the two spreads equal and press “New draw” several times: PC1's direction is unstable.
Turn on projection residuals and confirm they meet PC1 at right angles, not vertically.
Set the minor spread near zero: PC1 explains almost everything, because the cloud really is one-dimensional.

Common misconception

“PCA finds the intrinsic structure of my data, whatever units I measure in.”

PCA on the covariance matrix is not invariant to rescaling a single variable. Changing metres to millimetres inflates that variable's variance by a factor of a million and drags PC1 onto it. Only decomposition of the correlation matrix (standardizing first) is invariant, which is why it is the default for variables in different units.

DSPA3 Chapter 4 — Dimensionality reduction