Recap of Lecture 27 and Homework Check
LAST LECTURE · 5 MIN
Today: an MLP sees one input and forgets it. How can a network remember yesterday's electricity use, or the start of a sentence?
Today's Plan (50 minutes)
ROADMAP
0-5
Recap of Lecture 27 and homework check
S02
5-8
Plan and learning outcomes
S03-S04
8-16
Module 1: Sequences and the recurrent neuron
S05-S07
16-27
Module 2: Memory, training through time, and LSTM
S08-S11
27-35
Module 3: An RNN on real electricity data
S12-S14
35-43
Class activity: RNN steps by hand
S15
43-48
Common mistakes and recap quiz
S16-S17
48-50
Homework and next lecture
S18
Learning Outcomes (SLOs)
BY THE END OF THIS LECTURE YOU WILL BE ABLE TO
SLO 28.1
Explain why sequence data needs memory and describe the structure of a recurrent neural network
→ CLO3 · C2 Understand
SLO 28.2
Compute the hidden state of a simple RNN over a short input sequence
→ CLO3 · C3 Apply
SLO 28.3
Explain the vanishing gradient problem and how LSTM gates address it
→ CLO3 · C2 Understand
- Course Learning Outcome(s) served
- CLO3: Apply various techniques and algorithms of AI used in general problem solving, optimization problems, constraint satisfaction problems, and game programming.
Slide 5
MODULE 1
Sequences and the Recurrent Neuron
When order matters
Data That Comes in Order
MODULE 1 · SEQUENCES
Electricity
Language
Speech
Units day after day (P06b).
Words in a sentence; translation.
Sound samples -> Urdu text.
REAL OUTPUT: L28/SENTENCE.PY
- mehnat ki pass final h = [0.397 0.731]
- pass ki mehnat final h = [0.659 0.342]
- mehnat nahi ki final h = [0.099 0.606]
- same W and U for every word: 8 weights
Same words, different order: different final memory - the RNN 'notices' order.
The Recurrent Neuron and Unrolling
MODULE 1 · STRUCTURE
SIMPLE RNN (ELMAN, 1990)
- h_t = tanh( W·x_t + U·h_(t-1) + b )
- y_t = V·h_t + c
Unrolled: one copy per time step - but the SAME W, U, V at every step. Any length, fixed weights.
Picture: reading a novel - a running summary updated page by page.
Slide 8
MODULE 2
Memory, Training and LSTM
How long can an RNN remember?
Watching the Memory Fade
MODULE 2 · ONE NEURON BY HAND
REAL OUTPUT: L28/MEMORY.PY (W = 0.8, INPUT 1, 0, 0, 0, 0)
- u = 0.5: h = 0.664, 0.320, 0.159, 0.079, 0.040
- u = 0.9: h = 0.664, 0.535, 0.448, 0.382, 0.331
- by hand (u = 0.5): h1 = tanh(0.8*1 + 0.5*0) = 0.664; h2 = tanh(0.8*0 + 0.5*0.664) = 0.320
BY HAND (U = 0.5)
- h1 = tanh(0.8·1 + 0.5·0) = tanh(0.8) = 0.664
- h2 = tanh(0.8·0 + 0.5·0.664) = tanh(0.332) = 0.320
- h3 = tanh(0.8·0 + 0.5·0.320) = tanh(0.160) = 0.159
Key idea: the recurrent weight u controls how long the network remembers an event.
Training Through Time and the Vanishing Gradient
MODULE 2 · TRAINING
REAL OUTPUT: L28/VANISH.PY (ERROR SIGNAL AFTER K STEPS BACK = U^K)
- steps: 1 5 10 20
- u = 0.5: 0.5 0.0312 0.000977 9.54e-07
- u = 0.9: 0.9 0.59 0.349 0.122
- u = 1.5: 1.5 7.59 57.7 3.33e+03
Vanishing
u < 1: signal shrinks to almost 0 - long links are not learned.
BACK-PROPAGATION THROUGH TIME
Exploding
- BPTT: 1 unroll the network over the sequence
- 2 back-propagate through every copy
- 3 add up the updates for the shared W, U, V
u > 1: signal blows up - fix with gradient clipping.
Picture: a whisper passed along 20 students - it fades away (vanishing) or becomes a shout (exploding).
Described by Hochreiter (1991) and Bengio, Simard and Frasconi (1994).
LSTM: Gates That Decide What to Remember
MODULE 2 · LSTM AND GRU
Forget gate
Input gate
Output gate
What old memory to drop. 0 = forget, 1 = keep.
What new information to write into memory.
What part of memory to use for the output now.
Cell state: a conveyor belt that carries memory forward by ADDING, not multiplying - so the error signal survives many steps.
Slide 12
MODULE 3
An RNN on Real Data
Forecasting tomorrow's electricity
An RNN Forecasts Electricity Use
MODULE 3 · REAL DATA
REAL OUTPUT: L28/FORECAST.PY
- days: 730, test: last 120 days
- hidden neurons: 50
- MAE naive (tomorrow = today): 1.76 units
- MAE linear on 7 lags : 1.56 units
- MAE RNN (trained output) : 1.64 units
THE RECURRENT LOOP
Honest result: RNN 1.64 beats naive 1.76, but 7 lags (1.56) is as good on this simple series. Always compare with a baseline.
- for xt in x: # one day at a time
- h = np.tanh(W * xt + U @ h) # memory update
- # then: units tomorrow = V · [h, today, 1] (V by least squares)
Only V is trained here (an echo state network); full RNNs train W and U with BPTT in Keras or PyTorch.
Where Recurrent Networks Are Used
MODULE 3 · APPLICATIONS
Since 2017: Transformers (attention over all words at once) replaced RNNs in most language tasks - ChatGPT, Gemini and Claude are Transformers.
RNNs and LSTMs are still used on small devices and for streaming sensor data. Lecture 30 continues with NLP.
Activity: RNN Steps by Hand
CLASS ACTIVITY · PAIRS · 8 MIN
Pairs, 6 min + 2 min check. One-neuron RNN h_t = tanh(w·x_t + u·h_(t-1)), h_0 = 0. tanh(0.8) = 0.664, tanh(1.132) = 0.812.
w = 0.8, u = 0.5, inputs 1, 1. Compute h1 and h2.
Why can an RNN read a sentence of any length with a fixed number of weights?
In the formula, which weight carries the memory: W, U or V?
RNN with 3 inputs, 4 hidden neurons, 1 output: how many weights and biases?
Error signal factor u = 0.5 after 10 steps back? u = 1.5?
Which LSTM gate decides to drop old memory when a new sentence starts?
Common Mistakes
WATCH OUT
WRONG IDEA
CORRECT IDEA
✗ Each time step has its own weights.
✓ The SAME W, U, V are shared by all steps.
✗ An RNN remembers everything forever.
✓ Simple RNN memory fades; LSTM keeps it longer.
✗ Vanishing gradient = the output becomes 0.
✓ The ERROR SIGNAL becomes tiny, so early steps do not learn.
✗ A fancier model always beats a simple one.
✓ RNN 1.64 vs 7-lag linear 1.56: compare with baselines.
✗ ChatGPT is an RNN.
✓ It is a Transformer (attention, 2017).
Recap Quiz
CHECK YOUR UNDERSTANDING · 3 MIN
- Q1. The recurrent weights U connect:
- (a) Input to hidden (b) Hidden to output (c) Hidden at t-1 to hidden at t (d) Output to input
- Q2. w = 0.8, u = 0.5, input 1 then 0, h0 = 0. h2 is:
- (a) 0.664 (b) 0.320 (c) 0.800 (d) 0
- Q3. Training an RNN by unrolling it and back-propagating is called:
- (a) BPTT (b) ID3 (c) Q-learning (d) Resolution
- Q4. LSTM was proposed to reduce the:
- (a) Learning rate (b) Number of inputs (c) Overfitting of trees (d) Vanishing gradient problem
Homework and Next Lecture
BEFORE THE NEXT CLASS
- Task 1: One-neuron RNN with w = 0.6, u = 0.7, h0 = 0 reads the inputs 1, 0, 1. Compute h1, h2, h3 (use a calculator for tanh).
- Task 2: Error signal factor u = 0.8: compute 0.8^5, 0.8^10 and 0.8^30. What does this say about learning links 30 steps apart?
- Task 3: The forecast RNN has 1 input, 50 hidden neurons and 1 output. Count W, U, b, V and c. Why is U the biggest?
NEXT LECTURE
- Lecture 29:
- Genetic Algorithms