Nexus Learn

Lecture 27: Multilayer Networks and Back-propagation

Lecture Slides CS-511 Artificial Intelligence 7 min read Updated 8 Nov 2026

↓ Download slides (PowerPoint)

Recap of Lecture 26 and Homework Check

LAST LECTURE · 5 MIN

Today: XOR needs a hidden layer - but nobody knows the 'right answer' for a hidden neuron. How can it learn?

Today's Plan (50 minutes)

ROADMAP

0-5

Recap of Lecture 26 and homework check

S02

5-8

Plan and learning outcomes

S03-S04

8-16

Module 1: Layers and smooth neurons - the forward pass

S05-S07

16-27

Module 2: Back-propagation - one backward pass by hand

S08-S10

27-35

Module 3: Training a network - XOR, random starts, scikit-learn

S11-S14

35-43

Class activity: forward pass, deltas and updates

S15

43-48

Common mistakes and recap quiz

S16-S17

48-50

Homework and next lecture

S18

Learning Outcomes (SLOs)

BY THE END OF THIS LECTURE YOU WILL BE ABLE TO

SLO 27.1

Explain the structure of a multilayer network and compute its forward pass with sigmoid neurons

→ CLO3 · C3 Apply

SLO 27.2

Apply back-propagation to compute output and hidden deltas and update the weights once

→ CLO3 · C3 Apply

SLO 27.3

Explain how training settings (epochs, learning rate, hidden neurons, random start) affect learning

→ CLO3 · C2 Understand

  • Course Learning Outcome(s) served
  • CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.

Slide 5

MODULE 1

Layers and Smooth Neurons

The forward pass

From Step to Sigmoid

MODULE 1 · ACTIVATION

SIGMOID (LOGISTIC)

REAL OUTPUT: L27/SIGMOID_TABLE.PY

  • σ(z) = 1 / (1 + e^(-z))
  • slope = σ(z) · (1 - σ(z))
  • z step sigmoid slope s(1-s)
  • -4 0 0.018 0.018
  • -2 0 0.119 0.105
  • -1 0 0.269 0.197
  • 0 1 0.500 0.250
  • 1 1 0.731 0.197
  • 2 1 0.881 0.105
  • 4 1 0.982 0.018

Why it matters: a slope tells every weight which way to move to reduce the error - the step has none.

Other activations: tanh (-1 to 1, Lecture 28), ReLU = max(0, z) (deep learning; MLPClassifier's default).

A 2-2-1 Network and Its Forward Pass

MODULE 1 · FORWARD PASS

REAL OUTPUT: L27/FORWARD.PY (X = (1, 0), T = 1)

  • h1: z = 0.2*1 + (-0.3)*0 + 0.1 = 0.300
  • h1 = sigmoid(0.300) = 0.574
  • h2: z = 0.4*1 + 0.1*0 + (-0.2) = 0.200
  • h2 = sigmoid(0.200) = 0.550
  • y: z = 0.5*0.574 + (-0.4)*0.550 + 0.1 = 0.167
  • y = sigmoid(0.167) = 0.542
  • E = 1/2 (1 - 0.542)^2 = 0.1050

Hidden layer

Output layer

Error

h = σ(W·x + b)

y = σ(v·h + c)

E = ½ (t - y)² = 0.1050

Parameters: 2x2 weights + 2 biases + 2 weights + 1 bias = 9.

Slide 8

MODULE 2

Back-propagation

Sending the error backwards

Learning = Walking Downhill on the Error

MODULE 2 · THE IDEA

BACK-PROPAGATION FOR ONE EXAMPLE

  • 1 Forward: compute h, y and the error E.
  • 2 Backward: output delta, then hidden deltas through v.
  • 3 Update: every weight moves a little downhill.
  • output delta: d_o = (y - t) · y · (1 - y)
  • hidden delta: d_h = d_o · v · h · (1 - h)
  • update: weight <- weight - lr · delta · input
  • bias <- bias - lr · delta

Key idea: hidden neurons have no target - they receive a share of the output's error through v.

One Backward Pass by Hand

MODULE 2 · WORKED EXAMPLE

REAL OUTPUT: L27/BACKWARD.PY (X = (1, 0), T = 1, LR = 0.5)

  • output delta d_o = (y - t) y (1 - y) = (0.542 - 1)(0.542)(0.458) = -0.1138
  • hidden deltas d_h = d_o * v * h(1 - h) = -0.0139, 0.0113
  • new output weights v = 0.5327, -0.3687 bias c = 0.1569
  • new hidden weights h1 = (0.2070, -0.3000) b 0.1070; h2 = (0.3944, 0.1000) b -0.2056
  • output 0.5417 -> 0.5654; error 0.1050 -> 0.0945

THREE NUMBERS ON THE BOARD

  • d_o < 0: output too low - weights on active paths go up.
  • x2 = 0: its weights do not change.
  • Error: 0.1050 -> 0.0945 after one step.
  • v1 = 0.5 - 0.5 · (-0.1138) · 0.574 = 0.5327
  • d_h1 = -0.1138 · 0.5 · 0.574 · 0.426 = -0.0139
  • w(x1 -> h1) = 0.2 - 0.5 · (-0.0139) · 1 = 0.2070

Training: repeat forward + backward for every example, for thousands of epochs, until the error is small.

Slide 11

MODULE 3

Training a Network

XOR at last - and the practical settings

Back-propagation Learns XOR

MODULE 3 · NUMPY TRAINING

REAL OUTPUT: L27/XOR_TRAIN.PY

  • epoch 1: error 0.5183
  • epoch 500: error 0.0135
  • epoch 1000: error 0.0041
  • epoch 2000: error 0.0016
  • epoch 5000: error 0.0006
  • outputs: [0.017 0.981 0.984 0.015] -> [0 1 1 0] target [0 1 1 0]

THE TRAINING LOOP (L27/XOR_TRAIN.PY)

  • H = sigmoid(X @ W.T + b); Y = sigmoid(H @ V + c) # forward
  • d_o = (Y - T) * Y * (1 - Y); d_h = np.outer(d_o, V) * H * (1 - H) # backward
  • V -= lr * H.T @ d_o; W -= lr * d_h.T @ X # update (and the biases)

Result: outputs 0.017, 0.981, 0.984, 0.015 -> 0, 1, 1, 0 = XOR.

Random Starts, Local Minima and Settings

MODULE 3 · PRACTICAL ISSUES

REAL OUTPUT: L27/SEEDS.PY

  • seed 0: error 0.0006 learned
  • seed 1: error 0.2509 STUCK
  • seed 2: error 0.0006 learned
  • seed 3: error 0.0006 learned
  • seed 4: error 0.3342 STUCK
  • seed 5: error 0.0006 learned
  • seed 6: error 0.2508 STUCK
  • seed 7: error 0.0006 learned
  • seed 8: error 0.0006 learned
  • seed 9: error 0.0006 learned
  • 7 of 10 starts learned XOR

Local minimum: 3 of 10 starts got stuck - a valley that is not the lowest (like hill climbing, L15).

Fixes: several random starts, more hidden neurons, better optimisers (adam, lbfgs), momentum.

Hyperparameters = settings WE choose; parameters (weights, biases) are LEARNED.

The Same in scikit-learn

MODULE 3 · MLPCLASSIFIER

CODE: L27/MLP_SKLEARN.PY

REAL OUTPUT

  • from sklearn.neural_network import MLPClassifier
  • m = MLPClassifier(hidden_layer_sizes=(4,),
  • activation="logistic", solver="lbfgs",
  • random_state=0, max_iter=2000)
  • m.fit(X, T); m.predict(X)
  • XOR predictions: [0 1 1 0]
  • weight shapes: (2, 4) (4, 1)
  • moons, 400 points:
  • 1 hidden: train 0.86 test 0.90
  • 2 hidden: train 0.86 test 0.91
  • 5 hidden: train 0.94 test 0.95
  • 20 hidden: train 0.97 test 0.92

Best size: 5 hidden neurons (test 0.95); 20 fit training better but test worse - choose by test accuracy.

Activity: Forward Pass, Deltas and Updates

CLASS ACTIVITY · PAIRS · 8 MIN

Pairs, 6 min + 2 min check. Use the 2-2-1 network of S07 (weights on the board). σ(-0.2) = 0.450, σ(-0.1) = 0.475, σ(0.135) = 0.534.

σ(0) and its slope; σ(2)?

v = 0.5, lr = 0.5, d_o = -0.032, h = 0.6. New v?

Forward pass for x = (0, 1): h1, h2 and y.

Why did 3 of the 10 random starts fail to learn XOR?

y = 0.8, t = 1. Output delta d_o = (y - t) y (1 - y)?

How many parameters (weights + biases) does a 2-4-1 network have?

Common Mistakes

WATCH OUT

WRONG IDEA

CORRECT IDEA

✗ Back-propagation runs the network backwards to predict.

✓ Prediction is forward; only the ERROR goes backwards.

✗ Hidden neurons are given target values.

✓ They get a share of the output delta through v.

✗ Update: w <- w + lr · delta · input.

✓ Minus: move DOWNHILL on the error.

✗ One training run always finds the best weights.

✓ It can stick in a local minimum; try several starts.

✗ More hidden neurons is always better.

✓ Too many can overfit - check test accuracy.

Recap Quiz

CHECK YOUR UNDERSTANDING · 3 MIN

  • Q1. The slope of the sigmoid at z = 0 is:
  • (a) 0 (b) 0.5 (c) 0.25 (d) 1
  • Q2. Back-propagation computes the error slopes using the:
  • (a) Chain rule (b) Bayes' rule (c) Resolution rule (d) Perceptron rule
  • Q3. y = 0.5, t = 1. The output delta (y - t) y (1 - y) is:
  • (a) 0.125 (b) -0.125 (c) -0.5 (d) 0.25
  • Q4. Training settings we choose ourselves are called:
  • (a) Weights (b) Biases (c) Deltas (d) Hyperparameters

Homework and Next Lecture

BEFORE THE NEXT CLASS

  • Task 1: Forward pass of the S07 network for x = (1, 1), t = 0: h1, h2, y and E.
  • Task 2: For the same example do one backward pass with lr = 0.5: d_o, the two hidden deltas, the new v1, v2 and c.
  • Task 3: P05a's digits network has 64 inputs, 32 hidden neurons and 10 outputs. How many weights and biases?

NEXT LECTURE

  • Lecture 28:
  • Recurrent Neural Networks

← All CS-511 material