Backpropagation, One Edge at a Time

A tiny network with every forward value and every gradient shown on the graph.

Ch 14 14.2

Animator

Deep Learning

25 mindifficulty 4/5

Controls

Hidden activation

Hidden layers2

Units per hidden layer3

Training example (x1, x2 → y)

Learning rateη0.800

Steps to train20

Computational graph

Stage 1 / 7: forward through layer 1

forward pass (active)backward pass (active)selected edge

Loss for this example: not yet computed — step forward to the output. Click any edge to inspect its weight and gradient below.

Training loss

0 gradient-descent steps taken so far (full-batch).

Gradient magnitude by layer

Mean |∂L/∂w| per weight matrix, this example — watch it shrink toward the input with sigmoid and depth.

Finite-difference check

Central-difference numerical gradient versus the analytic backprop gradient for every edge in one layer.

edge	analytic ∂L/∂w	finite-difference	\|difference\|
unit 0 → unit 0	0	0	0
unit 1 → unit 0	-0.00160	-0.00160	1.27e-12
unit 0 → unit 1	0	0	0
unit 1 → unit 1	-0.00192	-0.00192	1.45e-12
unit 0 → unit 2	0	0	0
unit 1 → unit 2	0.00650	0.00650	4.73e-12

Output activation ŷ

0.4082

target y = 1

Loss, this example

0.1751

Mean training loss

—

0 steps

Weight matrices

2 → 3 → 3 → 1

Backpropagation is not the learner

Backpropagation is only the chain rule applied edge by edge through the graph: it produces every ∂L/∂w number shown above. It never decides how far to move a weight. That decision — subtract η times the gradient — is gradient descent, a separate step applied after backpropagation finishes. Swap gradient descent for Adam or RMSProp and every gradient computed here is unchanged; only the update rule differs.

Predict before you look

A sigmoid network has 3 hidden layers. Roughly how much smaller is a typical gradient reaching the first layer than one at the output layer?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A 2-input, 1-output fully connected network, 1-3 hidden layers deep, drawn as a computational graph. Every node shows its pre-activation z and activation a; every edge shows its weight and, once the backward pass reaches it, its exact ∂L/∂w. Stepping through forward, loss, and backward stages one at a time separates 'compute the derivative' (backpropagation) from 'move the weight' (gradient descent).

The chain rule, mechanically

Forward pass: for each layer l, z^(l) = W^(l) a^(l-1) + b^(l), then a^(l) = φ(z^(l)) for a chosen hidden activation φ, and a sigmoid output unit so the loss is comparable across activation choices.

Loss: L = ½(ŷ − y)² for the single training example currently selected.

Backward pass: define δ^(L) = (ŷ − y) ŷ(1 − ŷ) at the output, then propagate δ^(l) = (W^(l+1)ᵀ δ^(l+1)) ⊙ φ'(z^(l)) backward one layer at a time. Every weight's gradient is a local product: ∂L/∂W^(l) = δ^(l+1) (a^(l))ᵀ.

Because sigmoid's derivative a(1−a) never exceeds 0.25, each layer you push δ backward through multiplies it by at most 0.25 — after three sigmoid layers, gradients reaching the first layer can be under 1.6% of their size at the output. ReLU's derivative is 0 or 1, so it does not shrink surviving gradients at all.

z^(l) = W^(l) a^(l-1) + b^(l),  a^(l) = φ(z^(l))

δ^(L) = (ŷ − y)·ŷ(1 − ŷ)

δ^(l) = (W^(l+1)ᵀ δ^(l+1)) ⊙ φ'(z^(l))

∂L/∂W^(l) = δ^(l+1) (a^(l))ᵀ,   ∂L/∂b^(l) = δ^(l+1)

gradient descent: W ← W − η ∂L/∂W   (a separate rule, applied after ∂L/∂W is known)

Try this

Step through one example with sigmoid activation. Before revealing each backward stage, predict whether the next gradient will be larger or smaller than the one you just saw.
Compare the gradient-magnitude bar chart for sigmoid versus ReLU at 3 hidden layers. Which activation lets the first layer keep learning?
Click an edge near the input with sigmoid and 3 hidden layers, note its gradient, then click the corresponding edge near the output. Compare orders of magnitude.
Read the finite-difference table for a layer you have not touched. Confirm the analytic and numerical columns agree to several decimal places before trusting the rest of the graph.
Train 100 steps at η = 0.8 with 2 hidden layers, then push η to 4 and retrain from a fresh initialization. Same gradients, same architecture — only the update size changed.

Common misconception

“Backpropagation is the algorithm that trains a neural network.”

Backpropagation only computes ∂L/∂w for every weight, using the chain rule layer by layer. Training also requires an update rule — plain gradient descent, momentum, Adam, or anything else — that decides how to use those derivatives. The gradients in this lab do not change if you imagine using a different optimizer; only the weight-update arithmetic would.

Ch 13: Gradient Descent and Its Variants