Recap of Lecture 26 and Homework Check
LAST LECTURE · 5 MIN
Today: XOR needs a hidden layer - but nobody knows the 'right answer' for a hidden neuron. How can it learn?
Today's Plan (50 minutes)
ROADMAP
0-5
Recap of Lecture 26 and homework check
S02
5-8
Plan and learning outcomes
S03-S04
8-16
Module 1: Layers and smooth neurons - the forward pass
S05-S07
16-27
Module 2: Back-propagation - one backward pass by hand
S08-S10
27-35
Module 3: Training a network - XOR, random starts, scikit-learn
S11-S14
35-43
Class activity: forward pass, deltas and updates
S15
43-48
Common mistakes and recap quiz
S16-S17
48-50
Homework and next lecture
S18
Learning Outcomes (SLOs)
BY THE END OF THIS LECTURE YOU WILL BE ABLE TO
SLO 27.1
Explain the structure of a multilayer network and compute its forward pass with sigmoid neurons
→ CLO3 · C3 Apply
SLO 27.2
Apply back-propagation to compute output and hidden deltas and update the weights once
→ CLO3 · C3 Apply
SLO 27.3
Explain how training settings (epochs, learning rate, hidden neurons, random start) affect learning
→ CLO3 · C2 Understand
- Course Learning Outcome(s) served
- CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.
Slide 5
MODULE 1
Layers and Smooth Neurons
The forward pass
From Step to Sigmoid
MODULE 1 · ACTIVATION
SIGMOID (LOGISTIC)
REAL OUTPUT: L27/SIGMOID_TABLE.PY
- σ(z) = 1 / (1 + e^(-z))
- slope = σ(z) · (1 - σ(z))
- z step sigmoid slope s(1-s)
- -4 0 0.018 0.018
- -2 0 0.119 0.105
- -1 0 0.269 0.197
- 0 1 0.500 0.250
- 1 1 0.731 0.197
- 2 1 0.881 0.105
- 4 1 0.982 0.018
Why it matters: a slope tells every weight which way to move to reduce the error - the step has none.
Other activations: tanh (-1 to 1, Lecture 28), ReLU = max(0, z) (deep learning; MLPClassifier's default).
A 2-2-1 Network and Its Forward Pass
MODULE 1 · FORWARD PASS
REAL OUTPUT: L27/FORWARD.PY (X = (1, 0), T = 1)
- h1: z = 0.2*1 + (-0.3)*0 + 0.1 = 0.300
- h1 = sigmoid(0.300) = 0.574
- h2: z = 0.4*1 + 0.1*0 + (-0.2) = 0.200
- h2 = sigmoid(0.200) = 0.550
- y: z = 0.5*0.574 + (-0.4)*0.550 + 0.1 = 0.167
- y = sigmoid(0.167) = 0.542
- E = 1/2 (1 - 0.542)^2 = 0.1050
Hidden layer
Output layer
Error
h = σ(W·x + b)
y = σ(v·h + c)
E = ½ (t - y)² = 0.1050
Parameters: 2x2 weights + 2 biases + 2 weights + 1 bias = 9.
Slide 8
MODULE 2
Back-propagation
Sending the error backwards
Learning = Walking Downhill on the Error
MODULE 2 · THE IDEA
BACK-PROPAGATION FOR ONE EXAMPLE
- 1 Forward: compute h, y and the error E.
- 2 Backward: output delta, then hidden deltas through v.
- 3 Update: every weight moves a little downhill.
- output delta: d_o = (y - t) · y · (1 - y)
- hidden delta: d_h = d_o · v · h · (1 - h)
- update: weight <- weight - lr · delta · input
- bias <- bias - lr · delta
Key idea: hidden neurons have no target - they receive a share of the output's error through v.
One Backward Pass by Hand
MODULE 2 · WORKED EXAMPLE
REAL OUTPUT: L27/BACKWARD.PY (X = (1, 0), T = 1, LR = 0.5)
- output delta d_o = (y - t) y (1 - y) = (0.542 - 1)(0.542)(0.458) = -0.1138
- hidden deltas d_h = d_o * v * h(1 - h) = -0.0139, 0.0113
- new output weights v = 0.5327, -0.3687 bias c = 0.1569
- new hidden weights h1 = (0.2070, -0.3000) b 0.1070; h2 = (0.3944, 0.1000) b -0.2056
- output 0.5417 -> 0.5654; error 0.1050 -> 0.0945
THREE NUMBERS ON THE BOARD
- d_o < 0: output too low - weights on active paths go up.
- x2 = 0: its weights do not change.
- Error: 0.1050 -> 0.0945 after one step.
- v1 = 0.5 - 0.5 · (-0.1138) · 0.574 = 0.5327
- d_h1 = -0.1138 · 0.5 · 0.574 · 0.426 = -0.0139
- w(x1 -> h1) = 0.2 - 0.5 · (-0.0139) · 1 = 0.2070
Training: repeat forward + backward for every example, for thousands of epochs, until the error is small.
Slide 11
MODULE 3
Training a Network
XOR at last - and the practical settings
Back-propagation Learns XOR
MODULE 3 · NUMPY TRAINING
REAL OUTPUT: L27/XOR_TRAIN.PY
- epoch 1: error 0.5183
- epoch 500: error 0.0135
- epoch 1000: error 0.0041
- epoch 2000: error 0.0016
- epoch 5000: error 0.0006
- outputs: [0.017 0.981 0.984 0.015] -> [0 1 1 0] target [0 1 1 0]
THE TRAINING LOOP (L27/XOR_TRAIN.PY)
- H = sigmoid(X @ W.T + b); Y = sigmoid(H @ V + c) # forward
- d_o = (Y - T) * Y * (1 - Y); d_h = np.outer(d_o, V) * H * (1 - H) # backward
- V -= lr * H.T @ d_o; W -= lr * d_h.T @ X # update (and the biases)
Result: outputs 0.017, 0.981, 0.984, 0.015 -> 0, 1, 1, 0 = XOR.
Random Starts, Local Minima and Settings
MODULE 3 · PRACTICAL ISSUES
REAL OUTPUT: L27/SEEDS.PY
- seed 0: error 0.0006 learned
- seed 1: error 0.2509 STUCK
- seed 2: error 0.0006 learned
- seed 3: error 0.0006 learned
- seed 4: error 0.3342 STUCK
- seed 5: error 0.0006 learned
- seed 6: error 0.2508 STUCK
- seed 7: error 0.0006 learned
- seed 8: error 0.0006 learned
- seed 9: error 0.0006 learned
- 7 of 10 starts learned XOR
Local minimum: 3 of 10 starts got stuck - a valley that is not the lowest (like hill climbing, L15).
Fixes: several random starts, more hidden neurons, better optimisers (adam, lbfgs), momentum.
Hyperparameters = settings WE choose; parameters (weights, biases) are LEARNED.
The Same in scikit-learn
MODULE 3 · MLPCLASSIFIER
CODE: L27/MLP_SKLEARN.PY
REAL OUTPUT
- from sklearn.neural_network import MLPClassifier
- m = MLPClassifier(hidden_layer_sizes=(4,),
- activation="logistic", solver="lbfgs",
- random_state=0, max_iter=2000)
- m.fit(X, T); m.predict(X)
- XOR predictions: [0 1 1 0]
- weight shapes: (2, 4) (4, 1)
- moons, 400 points:
- 1 hidden: train 0.86 test 0.90
- 2 hidden: train 0.86 test 0.91
- 5 hidden: train 0.94 test 0.95
- 20 hidden: train 0.97 test 0.92
Best size: 5 hidden neurons (test 0.95); 20 fit training better but test worse - choose by test accuracy.
Activity: Forward Pass, Deltas and Updates
CLASS ACTIVITY · PAIRS · 8 MIN
Pairs, 6 min + 2 min check. Use the 2-2-1 network of S07 (weights on the board). σ(-0.2) = 0.450, σ(-0.1) = 0.475, σ(0.135) = 0.534.
σ(0) and its slope; σ(2)?
v = 0.5, lr = 0.5, d_o = -0.032, h = 0.6. New v?
Forward pass for x = (0, 1): h1, h2 and y.
Why did 3 of the 10 random starts fail to learn XOR?
y = 0.8, t = 1. Output delta d_o = (y - t) y (1 - y)?
How many parameters (weights + biases) does a 2-4-1 network have?
Common Mistakes
WATCH OUT
WRONG IDEA
CORRECT IDEA
✗ Back-propagation runs the network backwards to predict.
✓ Prediction is forward; only the ERROR goes backwards.
✗ Hidden neurons are given target values.
✓ They get a share of the output delta through v.
✗ Update: w <- w + lr · delta · input.
✓ Minus: move DOWNHILL on the error.
✗ One training run always finds the best weights.
✓ It can stick in a local minimum; try several starts.
✗ More hidden neurons is always better.
✓ Too many can overfit - check test accuracy.
Recap Quiz
CHECK YOUR UNDERSTANDING · 3 MIN
- Q1. The slope of the sigmoid at z = 0 is:
- (a) 0 (b) 0.5 (c) 0.25 (d) 1
- Q2. Back-propagation computes the error slopes using the:
- (a) Chain rule (b) Bayes' rule (c) Resolution rule (d) Perceptron rule
- Q3. y = 0.5, t = 1. The output delta (y - t) y (1 - y) is:
- (a) 0.125 (b) -0.125 (c) -0.5 (d) 0.25
- Q4. Training settings we choose ourselves are called:
- (a) Weights (b) Biases (c) Deltas (d) Hyperparameters
Homework and Next Lecture
BEFORE THE NEXT CLASS
- Task 1: Forward pass of the S07 network for x = (1, 1), t = 0: h1, h2, y and E.
- Task 2: For the same example do one backward pass with lr = 0.5: d_o, the two hidden deltas, the new v1, v2 and c.
- Task 3: P05a's digits network has 64 inputs, 32 hidden neurons and 10 outputs. How many weights and biases?
NEXT LECTURE
- Lecture 28:
- Recurrent Neural Networks