Nexus Learn

Lecture 28: Recurrent Neural Networks

Lecture Notes CS-511 Artificial Intelligence 5 min read Updated 8 Nov 2026

↓ Download notes (Word)

On this page
  1. Learning Outcomes
  2. 1. Sequences and the Recurrent Neuron
  3. 2. Memory, Training and LSTM
  4. 3. An RNN on Real Data
  5. 4. Common Mistakes
  6. Word Bank
  7. Self-Check Questions

Learning Outcomes

SLO

After this lecture you will be able to...

CLO

Bloom

SLO 28.1

Explain why sequence data needs memory and describe the structure of a recurrent neural network

CLO3

C2 Understand

SLO 28.2

Compute the hidden state of a simple RNN over a short input sequence

CLO3

C3 Apply

SLO 28.3

Explain the vanishing gradient problem and how LSTM gates address it

CLO3

C2 Understand

1. Sequences and the Recurrent Neuron

§1.1 Data that comes in order

Slide L28-S06

A sequence is data where order matters: daily electricity units (P06b), words in a sentence, sound samples in speech. An MLP takes a fixed-size input with no order - in P06b we had to choose 7 lag columns ourselves. A recurrent neural network (RNN) reads one item at a time and keeps a hidden state h (its memory) that is updated after every item.

MLP (L27)

RNN

Input

Fixed size, all at once

One item at a time, any length

Memory

None

Hidden state h

Order

Ignored unless we add lags

Built in

mehnat ki pass final h = [0.397 0.731]

pass ki mehnat final h = [0.659 0.342]

mehnat nahi ki final h = [0.099 0.606]

same W and U for every word: 8 weights

§1.2 The recurrent neuron and unrolling

Slide L28-S07

h_t = tanh( W·x_t + U·h_(t-1) + b ) W: input -> hidden

y_t = V·h_t + c U: hidden -> hidden (memory)

V: hidden -> output

Unrolling draws one copy of the network per time step. The same W, U and V are reused at every step, so an RNN reads sequences of any length with a fixed number of weights (Elman, 1990). Picture: reading a novel while keeping a running summary in your head.

2. Memory, Training and LSTM

§2.1 Watching the memory fade

Slide L28-S09

u = 0.5: h = 0.664, 0.320, 0.159, 0.079, 0.040

u = 0.9: h = 0.664, 0.535, 0.448, 0.382, 0.331

by hand (u = 0.5): h1 = tanh(0.8*1 + 0.5*0) = 0.664; h2 = tanh(0.8*0 + 0.5*0.664) = 0.320

With u = 0.5 the memory of the single input halves each step; with u = 0.9 it fades slowly. The recurrent weight u controls how long the network remembers, and training learns it from data.

§2.2 Training through time and the vanishing gradient

Slide L28-S10

Back-propagation through time (BPTT): unroll the network over the sequence, back-propagate (L27) through every copy, and add up the updates of the shared W, U, V. Each step back multiplies the error signal by about u (and by tanh's slope, at most 1):

steps: 1 5 10 20

u = 0.5: 0.5 0.0312 0.000977 9.54e-07

u = 0.9: 0.9 0.59 0.349 0.122

u = 1.5: 1.5 7.59 57.7 3.33e+03

u < 1: the signal vanishes, so links between items far apart are not learned (vanishing gradient - Hochreiter 1991; Bengio et al. 1994). u > 1: it explodes; the fix is gradient clipping. Picture: a whisper passed along 20 students.

§2.3 LSTM and GRU

Slide L28-S11

Gate (sigmoid, 0 to 1)

Job

Forget gate

What old memory to drop (e.g. a new sentence starts)

Input gate

What new information to write into memory

Output gate

What part of memory to use for the output now

Long Short-Term Memory (LSTM, Hochreiter and Schmidhuber, 1997) adds a cell state - a conveyor belt updated by adding, not by repeated multiplying - so the error signal survives many steps. The GRU (Cho et al., 2014) is a simpler version with two gates.

3. An RNN on Real Data

§3.1 Forecasting electricity use

Slide L28-S13

An RNN with 50 hidden tanh neurons reads the 730 days of household_series.csv one at a time. Its W and U are random and fixed and only the output layer is trained by least squares (like ANFIS in P11a) - an echo state network. Last 120 days = test.

days: 730, test: last 120 days

hidden neurons: 50

MAE naive (tomorrow = today): 1.76 units

MAE linear on 7 lags : 1.56 units

MAE RNN (trained output) : 1.64 units

The RNN beats the naive forecast without us choosing lags, but on this short, simple series a linear model on 7 lags is just as good. Always compare a new model with simple baselines.

§3.2 Where recurrent networks are used

Slide L28-S14

Speech to text, translation, next-word prediction, energy forecasting, handwriting. Since 2017 Transformers (attention over all words at once) have replaced RNNs in most language tasks - ChatGPT, Gemini and Claude are Transformers. RNNs are still used on small devices and for streaming data.

4. Common Mistakes

Wrong idea

Correct idea

Each time step has its own weights.

The same W, U, V are shared.

An RNN remembers everything forever.

Simple RNN memory fades; LSTM keeps it longer.

Vanishing gradient = output becomes 0.

The error signal becomes tiny.

A fancier model always wins.

Compare with baselines.

ChatGPT is an RNN.

It is a Transformer.

Word Bank

Word

Simple meaning

Example

Sequence

Data where order matters

daily units

Hidden state

The network's memory so far

h_t

Recurrent weight

Connects the old memory to the new one

U

Unrolling

One copy of the network per time step

3 days -> 3 copies

BPTT

Back-propagation through the unrolled network

train W, U, V

Vanishing gradient

Error signal shrinks to almost 0 going back

0.5^20

LSTM

RNN with gates and a cell state

Google Translate 2016

Gate

A 0-to-1 switch learned by a sigmoid

forget gate

Self-Check Questions

  1. Why did P06b need lag features but an RNN does not? (see §1.1)
  2. Write the RNN formula and name W, U, V. (see §1.2)
  3. w = 0.8, u = 0.9, inputs 1, 0: h1 and h2? (see §2.1)
  4. What is BPTT? (see §2.2)
  5. What do the three LSTM gates do? (see §2.3)
  6. Which model won on the electricity test data, and what is the lesson? (see §3.1)

Reading: NN Toolbox Guide Ch 9 (recurrent networks); Russell & Norvig Ch 21.6 (recurrent networks); P06b handout.

← All CS-511 material