Chapter 25: Recurrent Neural Networks (RNN), LSTM and GRU
MODULE 5 | WEEK 25
Chapter 25: Recurrent Neural Networks (RNN), LSTM and GRU
The data so far (tables, images) has been without any order . But in life a lot of data is ordered (a sequence) : the words of a sentence, audio, a share price, the weather. In it, what came earlier helps in understanding what comes later . In this chapter we will learn the RNN (in which the network has a memory ), its weakness (the vanishing gradient ), and its better forms LSTM and GRU , and we will see how text is fed into a network ( Embedding ).
25.1 What is sequence data?
A sequence is data in which order matters and the length can vary: sentences, a song, the daily weather reading, the price of a share, DNA. For an ordinary Dense network this creates three problems:
-
Order: "the dog bit the man" and "the man bit the dog" have the same words but the opposite meaning.
-
Different lengths: one sentence is 5 words, another is 50. A Dense layer needs a fixed input.
-
Earlier information: the meaning of the end of a sentence can depend on a word at the start ("My sister, who lives in Lahore, is coming today").
Let us see how Bag-of-Words (only counting words, not their order) loses the order:
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
sentences = ["dog bites man", "man bites dog"]
cv = CountVectorizer()
M = cv.fit_transform(sentences).toarray()
print("Vocabulary:", cv.get_feature_names_out())
print(M)
print("Do both sentences give the same numbers?", (M[0] == M[1]).all())
OUTPUT Vocabulary: ['bites' 'dog' 'man'] [[1 1 1] [1 1 1]] Do both sentences give the same numbers? True
Two sentences with completely different meanings , yet equal for Bag-of-Words. An RNN preserves the order.
25.2 RNN: the memory of a network
A Recurrent Neural Network reads a sequence one step (time step) at a time . At every step it takes the new input x_t along with the previous hidden state h_(t-1) (its own memory) and produces a new h_t . The same weights are reused at every step.
h_t = tanh( Wx * x_t + Wh * h_(t-1) + b )
x_t :
the input at this step
h_(t-1) :
the previous memory
h_t :
the new memory(goes on to the next step)
25.3 Practical forms of RNN
| Type | Input -> Output | Example |
|---|---|---|
| Many-to-One | The whole sequence -> one answer | Sentiment from a review (positive/negative) |
| One-to-Many | One input -> a sequence | A caption from an image |
| Many-to-Many (equal) | One output at every step | The part-of-speech of every word |
| Many-to-Many (encoder-decoder) | A sequence -> another sequence | Translation (English -> Urdu) |
For training, the network is unrolled in time : for a sequence of 4 steps it looks as if it were a 4-layer network whose weights are the same in all layers . Backpropagation runs on this unrolled network, and this is called Backpropagation Through Time (BPTT) .
25.4 The weakness of RNNs: Vanishing Gradient
In BPTT the gradient is sent backwards after being multiplied by Wh at every step . If Wh is "small", then over a long sequence the gradient dies out (the network forgets the early parts), and if it is big it blows up .
np.random.seed(1)
n = 20
Q, _ = np.linalg.qr(np.random.randn(n, n)) # a straight (orthogonal) matrix
for radius in [0.9, 1.0, 1.1]:
Wh = Q * radius # multiplies the size by 'radius' on every
turn
v = np.ones(n)
sizes = []
for t in range(1, 51):
v = Wh.T @ v # one step back
sizes.append(np.linalg.norm(v) / np.sqrt(n)) # compared to the starting size
print("Size of Wh", radius, " -> 10 steps back:", format(sizes[9], ".3f"), "times",
" | 50 steps back:", format(sizes[49], ".3f"), "times")
OUTPUT
Size of Wh 0.9 -> 10 steps back: 0.349 times | 50 steps back: 0.005 times Size of Wh 1.0 -> 10 steps back: 1.000 times | 50 steps back: 1.000 times Size of Wh 1.1 -> 10 steps back: 2.594 times | 50 steps back: 117.391 times When the effect of Wh is 0.9, 50 steps back the gradient is left at only 0.005 times the starting size ( vanishing ); at 1.1 it grows to 117 times ( exploding ). This is why a simple RNN cannot remember old information in a long sequence.
25.5 LSTM: long memory
LSTM (Long Short-Term Memory) has a separate cell state (C_t) that runs through the whole sequence like a "road", and three gates decide what to write on this road, what to erase and what to read:
| Gate | The question it asks | Result (0 to 1) |
|---|---|---|
| Forget gate (f) | How much of the old memory should Iforget? | 0 = forget everything, 1 = remember everything |
| Input gate (i) | How much of the new information should Iwrite? | 0 = nothing, 1 = all of it |
| Output gate (o) | How much of the cell state should Ishow? | 0 = nothing, 1 = all of it |
f = sigmoid( [x_t, h_(t-1)] @ Wf + bf ) forget gate
i = sigmoid( [x_t, h_(t-1)] @ Wi + bi ) input gate
o = sigmoid( [x_t, h_(t-1)] @ Wo + bo ) output gate
g = tanh ( [x_t, h_(t-1)] @ Wg + bg ) new candidate information
C_t = f * C_(t-1) + i * g cell state update
h_t = o * tanh(C_t) new hidden state
25.6 GRU: the lighter rival
GRU (Gated Recurrent Unit) is a simpler form of LSTM: no separate cell state , and only two gates : the update gate (how much of the old to keep, how much of the new to take) and the reset gate (how much of the previous memory to look at while computing the new one). Fewer parameters, often the same accuracy as LSTM, and faster training.
| Aspect | SimpleRNN | LSTM | GRU |
|---|---|---|---|
| Gates | None | 3 (+ cell state) | 2 |
| Long memory | Weak | Strong | Strong |
| Aspect | SimpleRNN | LSTM | GRU |
|---|---|---|---|
| Parameters | Fewest | Most | In between |
| Speed | Fastest | Slowest | Faster than LSTM |
| Typical use | For learning, short sequences | The default choice | When data is limited or speed is needed |
Counting parameters
SimpleRNN : (input + units + 1) x units LSTM : 4 x (input + units + 1) x units
GRU (Keras):
3 x (input x units + units x units + 2 x units) (Keras uses two sets of biases)
def rnn_p(inp, u):
return (inp + u + 1) * u
def lstm_p(inp, u):
return 4 * (inp + u + 1) * u
def gru_p(inp, u):
return 3 * (inp * u + u * u + 2 * u)
inp, units = 8, 16
print("SimpleRNN(16), input 8:", rnn_p(inp, units))
print("LSTM(16), input 8:", lstm_p(inp, units))
print("GRU(16), input 8:", gru_p(inp, units))
OUTPUT SimpleRNN(16), input 8: 400 LSTM(16), input 8: 1600 GRU(16), input 8: 1248
25.7 Feeding text into a network
A network understands only numbers, so we turn text into numbers in 3 steps: Tokenization (give each word a number), Padding (make all sentences the same length), and Embedding (give each word a list (vector) of learned numbers).
reviews = [
"the food was good",
"the service was bad",
"good food and good atmosphere",
"very bad food",
"great atmosphere and superb food",
"bad service and bad food",
]
labels = np.array([1, 0, 1, 0, 1, 0]) # 1 = positive, 0 = negative
# 1) Tokenization:
give every new word a number (0 is kept for padding)
vocab = {}
for r in reviews:
for w in r.split():
if w not in vocab:
vocab[w] = len(vocab) + 1
print("Vocabulary:", vocab)
encoded = [[vocab[w] for w in r.split()] for r in reviews]
print("Encoded :", encoded)
# 2) Padding:
bring every sentence to 5 words (zeros at the end)
max_len = 5
padded = np.array([seq + [0] * (max_len - len(seq)) for seq in encoded])
print("Padded:")
print(padded)
OUTPUT
Vocabulary: {'the': 1, 'food': 2, 'was': 3, 'good': 4, 'service': 5, 'bad': 6, 'and': 7,
'atmosphere': 8, 'very': 9, 'great': 10, 'superb': 11}
Encoded : [[1, 2, 3, 4], [1, 5, 3, 6], [4, 2, 7, 4, 8], [9, 6, 2], [10, 8, 7, 11, 2], [6,
5, 7, 6, 2]]
Padded:
[[ 1 2 3 4 0]
[ 1 5 3 6 0]
[ 4 2 7 4 8]
[ 9 6 2 0 0]
[10 8 7 11 2]
[ 6 5 7 6 2]]
The Embedding layer is a learnable table of shape (vocab_size, embedding_dim) . The number of each word picks one row of that table, and these rows are learned during training (words with similar meanings move closer together). It is much better than one-hot encoding, because it is small and rich in meaning .
vocab_size, emb_dim = len(vocab) + 1, 3 # +1 for padding
embedding = np.random.RandomState(0).randn(vocab_size, emb_dim) * 0.1 # random at the start
seq = padded[0] # "the food was good" + padding
vectors = embedding[seq] # the row of each word
print("Shape of one sentence: (words, embedding_dim) =", vectors.shape)
print("Shape of the embedding table:", embedding.shape)
print("Parameters of the embedding:", embedding.size)
OUTPUT
Shape of one sentence:
(words, embedding_dim) = (5, 3) Shape of the embedding table:
(12, 3) Parameters of the embedding:
36
25.8 RNN, LSTM, GRU in Keras
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential([
keras.Input(shape=(20,), dtype="int32"), # a sentence of 20 words (a list of numbers) layers.Embedding(input_dim=1000, output_dim=16, mask_zero=True), layers.LSTM(32), # many-to-one: the last hidden state layers.Dense(1, activation="sigmoid"), # positive/negative ]) model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"]) model.summary() OUTPUT (sample) Model: "sequential" ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃
Layer (type) Output Shape Param #
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ embedding (Embedding) │ (None, 20, 16) │ 16,000 │
│ lstm (LSTM) │ (None, 32) │ 6,272 │
│ dense (Dense) │ (None, 1) │ 33 │
└───────────────────────────────────┴──────────────────────────┴─────────────────┘
Total params: 22,305 (87.13 KB)
Trainable params: 22,305 (87.13 KB)
Non-trainable params: 0 (0.00 B)
return_sequences and stacking layers
If the next recurrent layer needs the output of every step , set return_sequences=True . A two-way ( Bidirectional ) layer reads the sentence both forwards and backwards (when the whole sentence is already available, like a review):
model2 = keras.Sequential([ keras.Input(shape=(20,), dtype="int32"), layers.Embedding(1000, 16, mask_zero=True), layers.Bidirectional(layers.GRU(32, return_sequences=True)), # output of every step layers.Bidirectional(layers.GRU(16)), # last layers.Dropout(0.3), layers.Dense(1, activation="sigmoid"), ])
model2.summary()
OUTPUT (sample) Model: "sequential" ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃ Layer (type) Output Shape Param # ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ embedding_1 (Embedding) │ (None, 20, 16) │ 16,000 │ │ bidirectional (Bidirectional) │ (None, 20, 64) │ 9,600 │ │ bidirectional_1 (Bidirectional) │ (None, 32) │ 7,872 │ │ dropout (Dropout) │ (None, 32) │ 0 │ │ dense_1 (Dense) │ (None, 1) │ 33 │ └───────────────────────────────────┴──────────────────────────┴─────────────────┘ Total params: 33,505 (130.88 KB) Trainable params: 33,505 (130.88 KB) Non-trainable params: 0 (0.00 B)
OUTPUT
Baseline predictions: [1 0 0] (1 = positive)
On small data the baseline works very well; the real benefit of an RNN/LSTM comes with big data and long sentences .
25.9 Common Mistakes and Hands-on
Hands-on
-
RNN by hand: change hidden_size in the NumPy RNN, count the parameters and check them against the formula.
-
Look at the gates: change the sequence in the LSTM NumPy code and see when the forget gate becomes large.
-
Parameters: work out the parameters of GRU(64) on input 32 (the reset_after formula).
-
Your own review dataset: write 20 sentences of your own (positive/negative) and build the TF-IDF + Logistic baseline, then an LSTM in Colab.
-
Experiment: replace LSTM with GRU in Keras and write down the comparison of parameters, time and accuracy.
KEY TAKEAWAYS
-
In sequence data the order matters; Bag-of-Words loses the order.
-
An RNN combines the input and the previous hidden state at every step into a new hidden state; the weights are shared across steps.
-
In a simple RNN the gradient vanishes/explodes over a long sequence. Clipping helps with exploding.
-
LSTM: long memory through the forget, input and output gates and the cell state. GRU: 2 gates, fewer parameters.
-
Parameters: SimpleRNN (in+u+1)u, LSTM 4 times that, GRU in Keras 3 x (inu + uu + 2u).
-
Text pipeline: Tokenization, Padding, Embedding. With mask_zero=True the padding is ignored.
-
return_sequences=True is for a following recurrent layer; Bidirectional takes double the parameters.
-
Always build a simple baseline first (TF-IDF + Logistic).