Gradient Descent: Step Size, Conditioning, and Momentum

Watch a convex bowl defeat gradient descent — the trouble is the condition number and the step size, not the landscape.

Ch 13 13.3

Animator

Optimization

18 mindifficulty 3/5

Controls

Optimizer

Learning rateη0.090

Stability needs η < 2/L = 0.100

Condition numberκ20

Ratio of the largest to the smallest curvature.

Momentumβ0.900

Iterations60

Start x-8

Start y6

Non-quadratic valley

A Rosenbrock-style curved ravine instead of the bowl.

The path over the landscape

Ellipses are level sets; the gradient is perpendicular to each of them.

trajectorystartminimumlevel sets

Loss per iteration

Linear scale, clipped at the start value.

Log loss per iteration

Straight line means geometric convergence.

Final loss

3.89e-4

start 392

Distance to optimum

0.02790

Iterations to 10⁻⁴ of start

within 60 steps

Stability bound 2/L

0.1000

η is inside the bound

Predicted rate (κ−1)/(κ+1)

0.9048

per-step contraction for optimal η

Status

converging

f(θ) = ½(θ₁² + κθ₂²)

Predict before you look

You run gradient descent on a strictly convex quadratic and the loss increases without bound. What is the most likely cause?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

Four optimizers on the same two-dimensional objective: a quadratic bowl whose condition number you set directly, or a curved Rosenbrock-style ravine. The trajectory is drawn over the level sets and the loss is shown on both a linear and a logarithmic axis, so convergence rate is visible as a slope.

Rates, steps, and curvature

For f(θ) = ½θᵗAθ with eigenvalues between m and L, gradient descent with step η contracts each eigen-direction by |1 − ηλ|. Stability therefore requires η < 2/L: exceed it and the largest-curvature direction amplifies every step, so a perfectly convex problem diverges.

The best possible step is η = 2/(m + L), which gives a per-step factor of (κ − 1)/(κ + 1) with κ = L/m. When κ = 200 that factor is 0.990, so thousands of iterations are needed — this is why ill-conditioned problems are slow even without any local minima.

Heavy-ball momentum accumulates a velocity, cancelling the oscillating component across the valley and improving the rate to roughly (√κ − 1)/(√κ + 1). Nesterov's variant evaluates the gradient at the look-ahead point, and RMSProp rescales each coordinate by a running gradient magnitude, which approximates a diagonal preconditioner.

θ₊ = θ − η∇f(θ),  contraction per direction = |1 − ηλ|

stability: η < 2/L,   optimal η = 2/(m + L)

GD rate = (κ−1)/(κ+1),   momentum rate ≈ (√κ−1)/(√κ+1)

RMSProp: sₜ = ρsₜ₋₁ + (1−ρ)g²,  θ₊ = θ − ηg/√(sₜ + ε)

Try this

With κ = 20 and gradient descent, raise η slowly past 2/L = 0.1 and watch a convex problem diverge.
At η = 0.09 compare the log-loss slope for gradient descent and momentum β = 0.9.
Set κ = 1: how many iterations does any method need?
Set κ = 200 with plain descent and 300 iterations. Does it arrive? Now add momentum.
Switch on the curved ravine and compare RMSProp with gradient descent from the same start.

Common misconception

“My loss exploded, so the loss surface must have a bad region or a saddle.”

Divergence on this lab's objective happens on a strictly convex quadratic with a unique minimum, purely because η exceeded 2/L. Step size interacts with the largest curvature, and no property of the landscape is needed to explain the failure. Symmetrically, slow progress on an ill-conditioned bowl looks like a plateau but is just the (κ−1)/(κ+1) rate — the fixes are scaling, preconditioning, and momentum, not a different loss function.

DSPA3 Chapter 13 — Optimization and numerical methods