SVD as an Optimal Summary

Rank-k truncation on an image and a data matrix, with the error bound made visible.

Ch 4 4.2

Explorer

Mathematical Core

20 mindifficulty 3/5

Controls

Data pattern

Rank keptk6

Number of singular components used to reconstruct the matrix.

Noise levelσ0.150

Additive Gaussian noise on every pixel — raise k too far and you start fitting this.

Log scale spectrum

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Original, rank-k reconstruction, and residual

k = 6 of 48 possible components.

Original (noisy)

Rank-6 reconstruction

Residual (original − reconstruction)

Compression ratio

25.3%

storage k(m+n+1)=582 vs mn=2304

Energy captured

95.8%

Relative Frobenius error

0.2043

sqrt(Σ_{i>k} σᵢ² / Σ σᵢ²) — the Eckart–Young bound, achieved exactly by truncated SVD.

Singular values used

Singular value spectrum

linear scale

Cumulative energy captured

Eckart–Young: truncated SVD is the optimal rank-k summary.

cumulative energy1 − (Eckart–Young error)²current k

More components is not always better

Each additional singular component reduces reconstruction error on the data you have, by construction — but once the true signal's few dominant components are captured, the rest of the spectrum is noise. Fitting it lowers training error while adding nothing (or something harmful) for new data. The elbow in the singular value spectrum, not the tail, is usually where the signal ends.

Predict before you look

You truncate an SVD from k=5 to k=10 components on a noisy image. What happens to the training-reconstruction error?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A synthetic image is decomposed by singular value decomposition, and truncating to the top k singular components gives the best possible rank-k approximation in Frobenius norm. The lab makes visible what that error bound (Eckart–Young) means, how compact a rank-k summary is compared to the raw pixel grid, and how adding noise changes which rank is worth keeping.

The Eckart–Young theorem

Every matrix A (n×m) admits a singular value decomposition A = UΣVᵀ with singular values σ₁ ≥ σ₂ ≥ ... ≥ 0 on the diagonal of Σ. Truncating to the top k terms, A_k = Σ_{i≤k} σᵢ uᵢ vᵢᵀ, is the closest rank-k matrix to A in both the spectral and Frobenius norms — no other rank-k matrix does better.

The Frobenius reconstruction error of the optimal rank-k approximation has a closed form: ‖A − A_k‖_F / ‖A‖_F = sqrt(Σ_{i>k} σᵢ² / Σ_i σᵢ²). This lab computes that quantity exactly and marks it on both the spectrum and the cumulative-energy plot.

This lab computes the SVD from the eigendecomposition of the Gram matrix AᵀA: its eigenvectors are the right singular vectors V, its eigenvalues are σᵢ², and the left singular vectors follow from U = AV/Σ. This mirrors how PCA and SVD are the same computation viewed from two ends.

A = UΣVᵀ,  A_k = argmin_{rank(B)=k} ‖A − B‖_F

‖A − A_k‖_F / ‖A‖_F = sqrt( Σ_{i>k} σᵢ² / Σ_i σᵢ² )

storage(A_k) = k(n + m + 1)  vs  storage(A) = nm

Try this

Set noise to 0 and slide k up from 0: watch the energy-captured curve jump quickly for the first few components on a low-rank pattern (like two blobs), then flatten.
Raise noise to 0.5+ and watch the singular value spectrum: a genuine signal still shows a few dominant values, but a noise floor of roughly-equal small values appears after it — that floor is where extra components stop helping.
Compare 'checker-gradient' (higher effective rank) to 'two-blobs' (very low effective rank) at the same k — the same compression ratio buys a much better reconstruction for the lower-rank pattern.
Toggle the log-scale spectrum to see the noise floor as a flat line — on a linear scale it can be invisible under the dominant singular values.

Common misconception

“Increasing the number of retained components always makes the summary better.”

It always makes the training reconstruction error weakly better, since each extra term can only add energy — but past the point where the retained components represent the true signal, the extra components are reconstructing noise. That is expensive (more storage per component) and does not generalize: it fits idiosyncrasies of this particular noisy draw, not the underlying pattern.

Compare with PCA as an eigen-decomposition of the covariance matrix