Learning Outcomes
SLO | After this lecture you will be able to... | CLO | Bloom |
|---|---|---|---|
SLO 28.1 | Explain why sequence data needs memory and describe the structure of a recurrent neural network | CLO3 | C2 Understand |
SLO 28.2 | Compute the hidden state of a simple RNN over a short input sequence | CLO3 | C3 Apply |
SLO 28.3 | Explain the vanishing gradient problem and how LSTM gates address it | CLO3 | C2 Understand |
1. Sequences and the Recurrent Neuron
§1.1 Data that comes in order
Slide L28-S06
A sequence is data where order matters: daily electricity units (P06b), words in a sentence, sound samples in speech. An MLP takes a fixed-size input with no order - in P06b we had to choose 7 lag columns ourselves. A recurrent neural network (RNN) reads one item at a time and keeps a hidden state h (its memory) that is updated after every item.
MLP (L27) | RNN | |
|---|---|---|
Input | Fixed size, all at once | One item at a time, any length |
Memory | None | Hidden state h |
Order | Ignored unless we add lags | Built in |
mehnat ki pass final h = [0.397 0.731] pass ki mehnat final h = [0.659 0.342] mehnat nahi ki final h = [0.099 0.606] same W and U for every word: 8 weights |
§1.2 The recurrent neuron and unrolling
Slide L28-S07
h_t = tanh( W·x_t + U·h_(t-1) + b ) W: input -> hidden y_t = V·h_t + c U: hidden -> hidden (memory) V: hidden -> output |
Unrolling draws one copy of the network per time step. The same W, U and V are reused at every step, so an RNN reads sequences of any length with a fixed number of weights (Elman, 1990). Picture: reading a novel while keeping a running summary in your head.
2. Memory, Training and LSTM
§2.1 Watching the memory fade
Slide L28-S09
u = 0.5: h = 0.664, 0.320, 0.159, 0.079, 0.040 u = 0.9: h = 0.664, 0.535, 0.448, 0.382, 0.331 by hand (u = 0.5): h1 = tanh(0.8*1 + 0.5*0) = 0.664; h2 = tanh(0.8*0 + 0.5*0.664) = 0.320 |
With u = 0.5 the memory of the single input halves each step; with u = 0.9 it fades slowly. The recurrent weight u controls how long the network remembers, and training learns it from data.
§2.2 Training through time and the vanishing gradient
Slide L28-S10
Back-propagation through time (BPTT): unroll the network over the sequence, back-propagate (L27) through every copy, and add up the updates of the shared W, U, V. Each step back multiplies the error signal by about u (and by tanh's slope, at most 1):
steps: 1 5 10 20 u = 0.5: 0.5 0.0312 0.000977 9.54e-07 u = 0.9: 0.9 0.59 0.349 0.122 u = 1.5: 1.5 7.59 57.7 3.33e+03 |
u < 1: the signal vanishes, so links between items far apart are not learned (vanishing gradient - Hochreiter 1991; Bengio et al. 1994). u > 1: it explodes; the fix is gradient clipping. Picture: a whisper passed along 20 students.
§2.3 LSTM and GRU
Slide L28-S11
Gate (sigmoid, 0 to 1) | Job |
|---|---|
Forget gate | What old memory to drop (e.g. a new sentence starts) |
Input gate | What new information to write into memory |
Output gate | What part of memory to use for the output now |
Long Short-Term Memory (LSTM, Hochreiter and Schmidhuber, 1997) adds a cell state - a conveyor belt updated by adding, not by repeated multiplying - so the error signal survives many steps. The GRU (Cho et al., 2014) is a simpler version with two gates.
3. An RNN on Real Data
§3.1 Forecasting electricity use
Slide L28-S13
An RNN with 50 hidden tanh neurons reads the 730 days of household_series.csv one at a time. Its W and U are random and fixed and only the output layer is trained by least squares (like ANFIS in P11a) - an echo state network. Last 120 days = test.
days: 730, test: last 120 days hidden neurons: 50 MAE naive (tomorrow = today): 1.76 units MAE linear on 7 lags : 1.56 units MAE RNN (trained output) : 1.64 units |
The RNN beats the naive forecast without us choosing lags, but on this short, simple series a linear model on 7 lags is just as good. Always compare a new model with simple baselines.
§3.2 Where recurrent networks are used
Slide L28-S14
Speech to text, translation, next-word prediction, energy forecasting, handwriting. Since 2017 Transformers (attention over all words at once) have replaced RNNs in most language tasks - ChatGPT, Gemini and Claude are Transformers. RNNs are still used on small devices and for streaming data.
4. Common Mistakes
Wrong idea | Correct idea |
|---|---|
Each time step has its own weights. | The same W, U, V are shared. |
An RNN remembers everything forever. | Simple RNN memory fades; LSTM keeps it longer. |
Vanishing gradient = output becomes 0. | The error signal becomes tiny. |
A fancier model always wins. | Compare with baselines. |
ChatGPT is an RNN. | It is a Transformer. |
Word Bank
Word | Simple meaning | Example |
|---|---|---|
Sequence | Data where order matters | daily units |
Hidden state | The network's memory so far | h_t |
Recurrent weight | Connects the old memory to the new one | U |
Unrolling | One copy of the network per time step | 3 days -> 3 copies |
BPTT | Back-propagation through the unrolled network | train W, U, V |
Vanishing gradient | Error signal shrinks to almost 0 going back | 0.5^20 |
LSTM | RNN with gates and a cell state | Google Translate 2016 |
Gate | A 0-to-1 switch learned by a sigmoid | forget gate |
Self-Check Questions
- Why did P06b need lag features but an RNN does not? (see §1.1)
- Write the RNN formula and name W, U, V. (see §1.2)
- w = 0.8, u = 0.9, inputs 1, 0: h1 and h2? (see §2.1)
- What is BPTT? (see §2.2)
- What do the three LSTM gates do? (see §2.3)
- Which model won on the electricity test data, and what is the lesson? (see §3.1)
Reading: NN Toolbox Guide Ch 9 (recurrent networks); Russell & Norvig Ch 21.6 (recurrent networks); P06b handout.