A guided tour of how a network learns. Each part explains one idea in a few lines, then lets you change the numbers and watch what happens. Read it before the lab, or come back to it when a training curve looks wrong.
Training repeats one loop: predict, measure the error, compute the gradient, update the parameters. The error is a single number called the loss. Everything in this resource is about making that loop converge faster and more reliably.
The learning rate η scales every update. Too small is slow, too large overshoots. The gradient itself is usually estimated from a mini-batch, which adds noise that shrinks as the batch grows.
In a narrow valley the gradient points mostly across it, so plain gradient descent zig-zags. Momentum keeps a decaying memory of earlier steps. Nesterov momentum checks the slope at the point momentum is about to reach.
AdaGrad, RMSProp and Adam divide the step by a running estimate of how large each parameter's gradients have been. Real networks do this for every parameter at once. Here one scalar gradient lets you see the internal state after each update.
A large learning rate makes quick early progress. A smaller one lets the weights settle. Weight decay adds a steady pull towards zero that discourages very large weights.
Features on very different scales stretch the loss surface into the narrow valley from Part 3. Standardising inputs, and normalising activations inside the network, rounds it out again.
When training goes wrong, form a hypothesis before changing the architecture. Use the table as a starting point, then test it with a controlled run.