Nexus Learn

Lecture 27: Multilayer Networks and Back-propagation

Lecture Notes CS-511 Artificial Intelligence 5 min read Updated 8 Nov 2026

↓ Download notes (Word)

On this page
  1. Learning Outcomes
  2. 1. Layers and Smooth Neurons
  3. 2. Back-propagation
  4. 3. Training a Network
  5. 4. Common Mistakes
  6. Word Bank
  7. Self-Check Questions

Learning Outcomes

SLO

After this lecture you will be able to...

CLO

Bloom

SLO 27.1

Explain the structure of a multilayer network and compute its forward pass with sigmoid neurons

CLO3

C3 Apply

SLO 27.2

Apply back-propagation to compute output and hidden deltas and update the weights once

CLO3

C3 Apply

SLO 27.3

Explain how training settings (epochs, learning rate, hidden neurons, random start) affect learning

CLO3

C2 Understand

1. Layers and Smooth Neurons

§1.1 From step to sigmoid

Slide L27-S06

The step function of the perceptron is flat (slope 0), so it cannot tell a weight which way to move. The sigmoid (logistic function) σ(z) = 1 / (1 + e^(-z)) is a smooth S-curve from 0 to 1 with slope σ(1 - σ) - largest (0.25) at z = 0. Picture: a light switch (step) against a dimmer (sigmoid). Other activations: tanh (-1 to 1) and ReLU = max(0, z).

z step sigmoid slope s(1-s)

-4 0 0.018 0.018

-2 0 0.119 0.105

-1 0 0.269 0.197

0 1 0.500 0.250

1 1 0.731 0.197

2 1 0.881 0.105

4 1 0.982 0.018

§1.2 A 2-2-1 network and its forward pass

Slide L27-S07

A multilayer perceptron (MLP) has an input layer, one or more hidden layers (not seen from outside; they build new features) and an output layer. The forward pass computes layer by layer: h = σ(W·x + b), then y = σ(v·h + c). Error for one example: E = ½ (t - y)².

h1: z = 0.2*1 + (-0.3)*0 + 0.1 = 0.300

h1 = sigmoid(0.300) = 0.574

h2: z = 0.4*1 + 0.1*0 + (-0.2) = 0.200

h2 = sigmoid(0.200) = 0.550

y: z = 0.5*0.574 + (-0.4)*0.550 + 0.1 = 0.167

y = sigmoid(0.167) = 0.542

E = 1/2 (1 - 0.542)^2 = 0.1050

Parameters of this network: 2 x 2 weights + 2 biases + 2 weights + 1 bias = 9.

2. Back-propagation

§2.1 Learning = walking downhill on the error

Slide L27-S09

Gradient descent moves every weight a small step against the slope of the error: w <- w - lr x slope (P11b used it for ANFIS). Back-propagation (Rumelhart, Hinton and Williams, 1986) finds all the slopes with the chain rule: first the output delta, then the hidden deltas, which receive a share of the output's error through the weights v.

output delta: d_o = (y - t) · y · (1 - y)

hidden delta: d_h = d_o · v · h · (1 - h)

update: weight <- weight - lr · delta · input bias <- bias - lr · delta

§2.2 One backward pass by hand

Slide L27-S10

output delta d_o = (y - t) y (1 - y) = (0.542 - 1)(0.542)(0.458) = -0.1138

hidden deltas d_h = d_o * v * h(1 - h) = -0.0139, 0.0113

new output weights v = 0.5327, -0.3687 bias c = 0.1569

new hidden weights h1 = (0.2070, -0.3000) b 0.1070; h2 = (0.3944, 0.1000) b -0.2056

output 0.5417 -> 0.5654; error 0.1050 -> 0.0945

d_o is negative because the output is too low, so the weights on active paths go up. The weights from x2 do not change because x2 = 0. After one update the error falls from 0.1050 to 0.0945.

3. Training a Network

§3.1 Back-propagation learns XOR

Slide L27-S12

epoch 1: error 0.5183

epoch 500: error 0.0135

epoch 1000: error 0.0041

epoch 2000: error 0.0016

epoch 5000: error 0.0006

outputs: [0.017 0.981 0.984 0.015] -> [0 1 1 0] target [0 1 1 0]

§3.2 Random starts, local minima and settings

Slide L27-S13

seed 0: error 0.0006 learned

seed 1: error 0.2509 STUCK

seed 2: error 0.0006 learned

seed 3: error 0.0006 learned

seed 4: error 0.3342 STUCK

seed 5: error 0.0006 learned

seed 6: error 0.2508 STUCK

seed 7: error 0.0006 learned

seed 8: error 0.0006 learned

seed 9: error 0.0006 learned

7 of 10 starts learned XOR

Three of ten random starts get stuck in a local minimum (a valley that is not the lowest). Fixes: several random starts, more hidden neurons, better optimisers (adam, lbfgs), momentum.

Hyperparameter (we choose)

Too small

Too large

Learning rate

Very slow

Jumps about

Epochs

Underfits

May overfit

Hidden neurons

Cannot bend

Memorises noise

§3.3 The same in scikit-learn

Slide L27-S14

from sklearn.neural_network import MLPClassifier

m = MLPClassifier(hidden_layer_sizes=(4,), activation="logistic", solver="lbfgs",

random_state=0, max_iter=2000).fit(X, T)

XOR predictions: [0 1 1 0]

weight shapes: (2, 4) (4, 1)

moons, 400 points:

1 hidden: train 0.86 test 0.90

2 hidden: train 0.86 test 0.91

5 hidden: train 0.94 test 0.95

20 hidden: train 0.97 test 0.92

Choose the number of hidden neurons by test accuracy: 5 is best here; 20 fits the training data better but the test data worse. P03 and P05 used MLPClassifier, P04 MLPRegressor - the same algorithm.

4. Common Mistakes

Wrong idea

Correct idea

Back-propagation predicts by running backwards.

Prediction is forward; only the error goes back.

Hidden neurons get targets.

They get a share of the output delta through v.

w <- w + lr · delta · input.

Minus: downhill.

One run always finds the best weights.

It can stick in a local minimum.

More hidden neurons is always better.

Too many can overfit.

Word Bank

Word

Simple meaning

Example

Sigmoid

Smooth S-shaped activation, 0 to 1

σ(0) = 0.5

Hidden layer

Neurons between input and output

h1, h2

Forward pass

Computing the output layer by layer

y = 0.542

Delta

How much the error changes with a neuron's input

d_o = -0.1138

Back-propagation

Sending deltas backwards to update all weights

Rumelhart et al., 1986

Local minimum

A valley that is not the lowest

3 of 10 XOR starts

Hyperparameter

A setting we choose

learning rate, hidden size

Self-Check Questions

  1. Why can the step function not be used with back-propagation? (see §1.1)
  2. Forward pass for x = (0, 0) in the S07 network. (see §1.2)
  3. Write the output and hidden delta formulas. (see §2.1)
  4. Why did the x2 weights not change in §2.2? (see §2.2)
  5. What is a local minimum and how can it be avoided? (see §3.2)
  6. How many parameters does a 3-5-2 network have? (see §3.3)

Reading: NN Toolbox Guide Ch 3, 5 (multilayer networks, back-propagation); Russell & Norvig Ch 21.2-21.4; P03-P05 handouts.

← All CS-511 material