The lab sheet runs in Google Colab. This page lets you play with each idea first: move the sliders, break the training, then run the same experiment in code.
Each unit multiplies its inputs by weights, adds a bias, then passes the result through an activation function. Change the numbers and watch z and h move.
Weights set how much each input matters. The bias shifts the whole curve, so the neuron fires earlier or later.
The output of one layer is the input of the next: h₁ = σ(W₁ᵀx + b₁), h₂ = σ(W₂ᵀh₁ + b₂), ŷ = softmax(W₃ᵀh₂ + b₃). Each 28×28 image is flattened into 784 numbers first, after dividing by 255 to land in [0, 1].
Parameters per Dense layer = inputs × units + units. The large model has about seven times more parameters than the baseline, and Task 8 trains it on only 1,000 images, split 700 for training and 300 for validation.
Backpropagation multiplies one derivative per layer. Sigmoid's derivative never exceeds 0.25, and it is close to zero once |z| is large. ReLU's derivative is exactly 1 for positive z.
Product of L slopes at the z you chose. Bars use a log scale.
The last Dense layer gives ten raw scores, one per digit. Softmax makes them positive and sums them to 1, so they read as probabilities. np.argmax then picks the predicted label, and the loss sparse_categorical_crossentropy is −log of the probability given to the true digit.
Gradient descent updates θ ← θ − η·∇L(θ). Here the loss is the simple bowl L(θ) = θ², starting at θ = 4, so the gradient is 2θ and each step multiplies θ by (1 − 2η). That makes the three behaviours easy to see.
In the lab you use Adam at η = 0.0001, 0.001 and 0.05. Adam rescales the step per parameter, so the exact thresholds differ from this toy bowl. The pattern is the same: very small learns slowly, moderate is smooth, large is noisy or unstable.
A model that is large relative to its data keeps improving on the training set while validation stops improving, then gets worse. This panel is a simulation that mimics typical curves. Your Colab run will look similar but noisier.
Read it like this: the gap between the two lines is the cost of memorising. The best epoch is where validation loss is lowest. Training further past it is where early stopping, regularisation or more data would help.
Open each card, copy the cell into Colab and run it in order. The note under each cell says what you should see.
Answers are saved in this browser only. Open a hint after you have tried.