MAI5063 Advances in Deep Learning · Week 2 companion lab

Why some runs converge and others do not

The same network can train fast, slowly, noisily or not at all. Only the optimiser, step size, batch size and schedule change. Explore each factor here, then run the controlled experiments on Fashion-MNIST in Colab.

Time 2 h (1.5 h guided) Platform Colab + TensorFlow/Keras Runtime GPU optional Data Fashion-MNIST, 28×28 grayscale

By the end you can

Build a configurable Keras network for image classification.
Compare SGD, SGD with momentum, RMSProp and Adam on the same model and data.
Explain how learning rate and batch size change convergence.
Use learning-rate scheduling and weight decay, and judge what they actually did.
Read curves as evidence of underfitting, overfitting, instability or good convergence.

Before the lab

Review gradient descent, mini-batch SGD, momentum, Nesterov momentum, AdaGrad, RMSProp, Adam, weight decay, scheduling and normalisation. Open colab.research.google.com and start a notebook, or upload the companion .ipynb. Run cells in order. A GPU is optional for this lab. Write answers in the reflection section.

Tasks 6 and 7 · Optimisers and learning rate

Four optimisers on the same bowl

Lab 1 showed step size on a one-dimensional bowl. Real losses are stretched: steep in some directions, shallow in others. Here the bowl is L = ½(x² + K·y²). Raise K and the contours squash into a narrow valley. All four optimisers start at the same point and use exact gradients, so every difference comes from the update rule.

SGD θ ← θ − η g Momentum u ← α u − η g, θ ← θ + u RMSProp s ← ρ s + (1 − ρ) g², θ ← θ − η g / (√s + ε) Adam m ← β₁ m + (1 − β₁) g, v ← β₂ v + (1 − β₂) g², θ ← θ − η m̂ / (√v̂ + ε)

Paths down the valley

Controls

Learning rate presets (K = 10)

Making the bowl rounder is what input scaling and normalisation layers do for you. At K = 1 every optimiser takes a straight path. At K = 10 plain SGD zig-zags across the valley, momentum builds speed along it, and RMSProp and Adam rescale each direction.

Task 8 · Mini-batch gradients

A batch gives a noisy estimate of the true gradient

Mini-batch SGD uses g = (1/B) Σ ∇ℓ over B examples instead of all 50,000. Each draw below is the mean gradient of one batch for a single weight. The spread of those means shrinks like σ/√B, but a larger batch also means fewer updates per epoch.

Forty batches at a time

Batch size B

batch mean gradienttrue gradient

What B changes

The table uses the lab's 50,000 training images. In Colab the batch-32 run does about 1,563 updates per epoch against 98 for batch 512, so it takes noticeably longer in wall-clock time.

Task 9 · Learning-rate scheduling

Large steps early, careful steps late

A high learning rate makes fast early progress but keeps the weights bouncing around the minimum. Lowering it later lets them settle. The lab halves the rate every three epochs. Change the schedule and compare with a constant rate.

Schedule

Validation loss, illustrative

with this scheduleconstant rate

A simulation: bounce scales with the current rate and the floor drops as the rate falls. Your Colab curves will be noisier, but the loss should settle after each reduction.

In Colab the step rule is step_decay(epoch, lr). It halves when the epoch index is 3, 6 and 9, so the 4th, 7th and 10th epochs are the first to use a lower rate.

Task 10 · Weight decay and normalisation

Check how much weight decay actually does

AdamW shrinks every weight by a factor of (1 − η·λ) at each step, on top of the gradient step. With the lab's η = 0.001 and λ = 0.0001 that factor is 0.9999999. Set the numbers below to see what that adds up to over a full run.

Shrinkage from decay alone

w ← w − η·(optimiser step) − η·λ·w

Learning rate η Weight decay λ

Total shrinkage by λ

Batch size 128 gives 391 updates per epoch. If AdamW is missing in your TensorFlow version, the lab falls back to plain Adam, which has no decay at all.

Batch Normalisation is the other half of Task 10. It re-centres and re-scales each layer's inputs, which keeps the loss surface closer to the round bowl above. The BatchNorm demo is in Lab 3, next to Dropout.

Tasks 5 to 10 · Predict, then run

Try the experiments before Colab

Choose a setup and watch the training and validation curves for the 256-128 network on Fashion-MNIST. The curves are a simulation tuned to typical behaviour, so your Colab numbers will differ. The ordering and the shapes should look familiar. The Adam baseline is drawn dashed for comparison.

Setup

Presets

Optimiser

Learning rate

Batch size

LR schedule

Extra

Plot

Training vs validation

trainingvalidationAdam baseline validation

Hands-on · Google Colab

The Colab tasks

Copy each cell into Colab and run them in order. Setup, loading and preprocessing are one card because they run together. Task numbers match the lab sheet. The Adam run in Task 6, the 0.001 run in Task 7 and the batch-128 run in Task 8 repeat the Task 5 baseline, so you can reuse its result if time is short.

Reflection

Write it in your own words

Answers are saved in this browser only. Open a hint after you have tried.