Chapter 25: Recurrent Neural Networks (RNN), LSTM aur GRU
MODULE 5 | WEEK 25
Chapter 25: Recurrent Neural Networks (RNN), LSTM aur GRU
Ab tak ka data (table, tasveer) tarteeb ke baghair tha. Magar zindagi me bohat sa data tarteeb wala (sequence) hota hai: jumlay ke alfaz, awaaz, share ki qeemat, mausam. Is me pehle kya aaya ye baad wale ko samajhne me madad deta hai. Is chapter me RNN (jis me network ki ek yaaddasht hoti hai), uski kamzori ( vanishing gradient ), aur uske behtar roop LSTM aur GRU seekhenge, aur dekhenge ke text ko network me kaise dalte hain ( Embedding ).
25.1 Sequence data kya hai?
Sequence woh data hai jis me tarteeb ahem hai aur lambai badal sakti hai: jumlay, gaana, mausam ki rozana reading, share ka bhao, DNA. Aam Dense network ke liye ye teen masail paida karta hai:
-
Tarteeb: "kutta insan ko kata" aur "insan ne kutte ko kata" me wohi alfaz hain magar matlab ulta.
-
Lambai alag alag: koi jumla 5 lafz ka, koi 50 ka. Dense layer ko fixed input chahiye.
-
Pichli maloomat: jumle ke aakhir ka matlab shuru ke lafz par nirbhar ho sakta hai ("meri behan jo Lahore me rehti hai, wo aaj aa rahi hai").
Dekhte hain ke Bag-of-Words (sirf alfaz ginna, tarteeb nahi) kaise tarteeb kho deta hai:
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
sentences = ["dog bites man", "man bites dog"]
cv = CountVectorizer()
M = cv.fit_transform(sentences).toarray()
print("Vocabulary:", cv.get_feature_names_out())
print(M)
print("Dono jumle ek jaise numbers?", (M[0] == M[1]).all())
OUTPUT
Vocabulary: ['bites' 'dog' 'man']
[[1 1 1]
[1 1 1]]
Dono jumle ek jaise numbers? True
Do bilkul mukhtalif matlab wale jumle, magar Bag-of-Words ke liye barabar . RNN tarteeb ko bachata hai.
25.2 RNN: network ki yaaddasht
Recurrent Neural Network sequence ko ek ek qadam (time step) par parhta hai. Har qadam par wo naye input x_t ke saath pichla hidden state h_(t-1) (apni yaaddasht) bhi leta hai aur naya h_t banata hai. Wohi weights har qadam par dobara istemal hote hain.
h_t = tanh( Wx * x_t + Wh * h_(t-1) + b )
x_t :
is qadam ka input
h_(t-1) :
pichli yaaddasht
h_t :
nayi yaaddasht (agle qadam ko jati hai)
OUTPUT
t = 0
h = [0.707 0.197 0.454 0.808]
t = 1
h = [ 0.934 -0.719 0.705 -0.278]
t = 2
h = [-0.896 0.646 0.434 0.742]
t = 3
h = [ 0.978 -0.668 0.79 0.37 ]
Weights ki shapes:
Wx (3, 4)
Wh (4, 4)
Parameters (kitni bhi lambi sequence ho):
32
25.3 RNN ke amli roop
| Qism | Input -> Output | Misaal |
|---|---|---|
| Many-to-One | Poori sequence -> ek jawab | Review se sentiment (positive/negative) |
| One-to-Many | Ek input -> sequence | Tasveer se caption |
| Many-to-Many (barabar) | Har qadam par ek output | Har lafz ka part-of-speech |
| Qism | Input -> Output | Misaal |
|---|---|---|
| Many-to-Many (encoder-decoder) | Sequence -> doosri sequence | Translation (Angrezi -> Urdu) |
Training ke liye network ko waqt me khol (unroll) diya jata hai: 4 qadam ki sequence ke liye aisa lagta hai jaise 4 layers ka network hai jis ke weights sab layers me barabar hain. Backpropagation isi unrolled network par chalta hai, isay Backpropagation Through Time (BPTT) kehte hain.
25.4 RNN ki kamzori: Vanishing Gradient
BPTT me gradient ko har qadam par Wh se guna kar ke peeche bheja jata hai. Agar Wh "chhota" ho to lambi sequence me gradient mar jata hai (network shuru ki baatein bhool jata hai), aur agar bara ho to phat jata hai.
np.random.seed(1)
n = 20
Q, _ = np.linalg.qr(np.random.randn(n, n)) # seedha (orthogonal) matrix
for radius in [0.9, 1.0, 1.1]:
Wh = Q * radius # har ghumao me size 'radius' guna
v = np.ones(n)
sizes = []
for t in range(1, 51):
v = Wh.T @ v # ek qadam peeche
sizes.append(np.linalg.norm(v) / np.sqrt(n)) # shuru ke size ke muqable me
print("Wh ka size", radius, " -> 10 qadam peeche:", format(sizes[9], ".3f"), "guna",
" | 50 qadam peeche:", format(sizes[49], ".3f"), "guna")
OUTPUT
Wh ka size 0.9 -> 10 qadam peeche: 0.349 guna | 50 qadam peeche: 0.005 guna
Wh ka size 1.0 -> 10 qadam peeche: 1.000 guna | 50 qadam peeche: 1.000 guna
Wh ka size 1.1 -> 10 qadam peeche: 2.594 guna | 50 qadam peeche: 117.391 guna
Jab Wh ka asar 0.9 ho, 50 qadam peeche gradient shuru ke size ka sirf 0.005 guna reh jata hai ( vanishing ); 1.1 par 117 guna barh jata hai ( exploding ). Is liye saada RNN lambi sequence me purani maloomat yaad nahi rakh pata.
25.5 LSTM: lambi yaaddasht
LSTM (Long Short-Term Memory) me ek alag cell state (C_t) hota hai, jo ek "sadak" ki tarah poori sequence se guzarta hai, aur teen gates (darwaze) tay karte hain ke is sadak par kya likhna, kya mitana aur kya parhna hai:
| Gate | Sawal jo wo poochta hai | Nateeja (0 se 1) |
|---|---|---|
| Forget gate (f) | Purani yaaddasht me se kitnabhoolun? | 0 = sab bhool, 1 = sab yaad |
| Input gate (i) | Nayi maloomat me se kitnalikhun? | 0 = kuch nahi, 1 = poora |
| Output gate (o) | Cell state me se kitnabahar dikhaun? | 0 = kuch nahi, 1 = poora |
f = sigmoid( [x_t, h_(t-1)] @ Wf + bf ) forget gate
i = sigmoid( [x_t, h_(t-1)] @ Wi + bi ) input gate
o = sigmoid( [x_t, h_(t-1)] @ Wo + bo ) output gate
g = tanh ( [x_t, h_(t-1)] @ Wg + bg ) nayi umeedwar maloomat
C_t = f * C_(t-1) + i * g cell state update
h_t = o * tanh(C_t) naya hidden state
OUTPUT
t = 0 | forget:
[0.45 0.49 0.26 0.69] | input:
[0.33 0.48 0.47 0.75] | output:
[0.42 0.56
0.53 0.28]
t = 1 | forget:
[0.27 0.37 0.57 0.32] | input:
[0.2 0.52 0.56 0.67] | output:
[0.4 0.48
0.54 0.49]
t = 2 | forget:
[0.35 0.39 0.56 0.73] | input:
[0.54 0.39 0.54 0.57] | output:
[0.21 0.5
0.45 0.31]
t = 3 | forget:
[0.25 0.41 0.28 0.54] | input:
[0.14 0.49 0.52 0.87] | output:
[0.38 0.55
0.51 0.38]
Aakhri hidden state h:
[-0.021 -0.078 0.139 0.27 ]
Aakhri cell state C:
[-0.056 -0.144 0.279 0.892]
LSTM parameters:
128
25.6 GRU: halka muqabil
GRU (Gated Recurrent Unit) LSTM ka saada roop hai: cell state alag nahi , aur sirf do gates : update gate (kitna purana rakhun, kitna naya lun) aur reset gate (pichli yaaddasht ka kitna hissa naya hisaab lagate waqt dekhun). Kam parameters, aksar LSTM jaisi accuracy, aur tez training.
| Pehlu | SimpleRNN | LSTM | GRU |
|---|---|---|---|
| Gates | Koi nahi | 3 (+ cell state) | 2 |
| Lambi yaaddasht | Kamzor | Mazboot | Mazboot |
| Parameters | Sab se kam | Sab se zyada | Beech me |
| Raftar | Sab se tez | Sab se dheemi | LSTM se tez |
| Aam istemal | Seekhne ke liye, chhoti sequence | Default pasand | Jab data kam ya raftar chahiye |
Parameters ki ginti
SimpleRNN : (input + units + 1) x units
LSTM : 4 x (input + units + 1) x units
GRU (Keras): 3 x (input x units + units x units + 2 x units) (Keras me do bias set hote
hain)
def rnn_p(inp, u): return (inp + u + 1) * u
def lstm_p(inp, u): return 4 * (inp + u + 1) * u
def gru_p(inp, u): return 3 * (inp * u + u * u + 2 * u)
inp, units = 8, 16
print("SimpleRNN(16), input 8:", rnn_p(inp, units))
print("LSTM(16), input 8:", lstm_p(inp, units))
print("GRU(16), input 8:", gru_p(inp, units))
OUTPUT SimpleRNN(16), input 8: 400 LSTM(16), input 8: 1600 GRU(16), input 8: 1248
25.7 Text ko network me dalna
Network sirf numbers samajhta hai, to text ko 3 qadmon me badalte hain: Tokenization (alfaz ko ginti de do), Padding (sab jumlon ko barabar lambai ka karo), aur Embedding (har lafz ko seekhe hue numbers ki list (vector) de do).
reviews = [
"khana bohat acha tha",
"service bohat kharab thi",
"acha khana aur acha mahaul",
"bohat kharab khana",
"mahaul acha tha khana lajawab",
"kharab service aur kharab khana",
]
labels = np.array([1, 0, 1, 0, 1, 0]) # 1 = positive, 0 = negative
# 1) Tokenization:
har naye lafz ko ginti (0 padding ke liye)
vocab = {}
for r in reviews:
for w in r.split():
if w not in vocab:
vocab[w] = len(vocab) + 1
print("Vocabulary:", vocab)
encoded = [[vocab[w] for w in r.split()] for r in reviews]
print("Encoded :", encoded)
# 2) Padding:
sab ko 5 lafz ki lambai tak (peeche zero)
max_len = 5
padded = np.array([seq + [0] * (max_len - len(seq)) for seq in encoded])
print("Padded:")
print(padded)
OUTPUT
Vocabulary: {'khana': 1, 'bohat': 2, 'acha': 3, 'tha': 4, 'service': 5, 'kharab': 6, 'thi':
7, 'aur': 8, 'mahaul': 9, 'lajawab': 10}
Encoded : [[1, 2, 3, 4], [5, 2, 6, 7], [3, 1, 8, 3, 9], [2, 6, 1], [9, 3, 4, 1, 10], [6,
5, 8, 6, 1]]
Padded:
[[ 1 2 3 4 0]
[ 5 2 6 7 0]
[ 3 1 8 3 9]
[ 2 6 1 0 0]
[ 9 3 4 1 10]
[ 6 5 8 6 1]]
Embedding layer ek seekhne wali table hai: shape (vocab_size, embedding_dim). Har lafz ki ginti us table ki ek row chunti hai, aur ye rows training me seekhi jati hain (milte julte matlab wale alfaz qareeb aate jate hain). Ye one-hot encoding se bohat behtar hai, kyun ke chhoti aur maani se bhari hoti hai.
vocab_size, emb_dim = len(vocab) + 1, 3 # +1 padding ke liye
embedding = np.random.RandomState(0).randn(vocab_size, emb_dim) * 0.1 # shuru me random
seq = padded[0] # "khana bohat acha tha" + padding
vectors = embedding[seq] # har lafz ki row
print("Ek jumle ki shape: (lafz, embedding_dim) =", vectors.shape)
print("Embedding table ki shape:", embedding.shape)
print("Embedding ke parameters:", embedding.size)
OUTPUT
Ek jumle ki shape: (lafz, embedding_dim) = (5, 3)
Embedding table ki shape: (11, 3)
Embedding ke parameters: 33
25.8 Keras me RNN, LSTM, GRU
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential([
keras.Input(shape=(20,), dtype="int32"), # 20 lafz ki jumla (ginti ki list)
layers.Embedding(input_dim=1000, output_dim=16, mask_zero=True),
layers.LSTM(32), # many-to-one: aakhri hidden state
layers.Dense(1, activation="sigmoid"), # positive/negative
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.summary()
OUTPUT (namoona)
Model: "sequential"
| Layer (type) | Output shape | Param # |
|---|---|---|
| embedding (Embedding) | (None, 20, 16) | 16,000 |
| lstm (LSTM) | (None, 32) | 6,272 |
| dense (Dense) | (None, 1) | 33 |
| Total params | 22,305 (87.13 KB) | |
| Trainable params | 22,305 (87.13 KB) | |
| Non-trainable params | 0 (0.00 B) |
return_sequences aur layers ko stack karna
Agar agli recurrent layer ko har qadam ka output chahiye to return_sequences=True lagao. Do tarfa ( Bidirectional ) layer jumle ko aage se aur peeche se dono taraf parhti hai (jab poora jumla pehle se maujood ho, jaise review):
model2 = keras.Sequential([
keras.Input(shape=(20,), dtype="int32"),
layers.Embedding(1000, 16, mask_zero=True),
layers.Bidirectional(layers.GRU(32, return_sequences=True)), # har qadam ka output
layers.Bidirectional(layers.GRU(16)), # aakhri
layers.Dropout(0.3),
layers.Dense(1, activation="sigmoid"),
])
model2.summary()
OUTPUT (namoona)
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type) Output Shape Param # ┃┃┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ embedding_1 (Embedding) │ (None, 20, 16) │ 16,000 │
│ bidirectional (Bidirectional) │ (None, 20, 64) │ 9,600 │
│ bidirectional_1 (Bidirectional) │ (None, 32) │ 7,872 │
│ dropout (Dropout) │ (None, 32) │ 0 │
│ dense_1 (Dense) │ (None, 1) │ 33 │
└───────────────────────────────────┴──────────────────────────┴─────────────────┘
Total params: 33,505 (130.88 KB)
Trainable params: 33,505 (130.88 KB)
Non-trainable params: 0 (0.00 B)
OUTPUT
Baseline predictions: [1 0 1] (1 = positive)
Chhote data par baseline bohat acha chal jata hai; RNN/LSTM ka asli faida bare data aur lambe jumlon par hota hai.
25.9 Aam Galtiyan aur Hands-on
AAM GALTIYAN
-
Padding aur masking bhoolna: network zero padding ko "lafz" samajh leta hai.
-
Sequences ko sort ya shuffle kar dena (waqt ki tarteeb bigad jati hai, khaaskar time series me).
-
Bohat lambi sequence par saada RNN (LSTM/GRU lagao).
-
Gradient clipping na lagana jab loss achanak
nanho jaye. -
Test data ka vocabulary training me shamil karna (leakage).
-
Bidirectional wahan lagana jahan agla lafz maloom nahi hota (jaise agle lafz ki prediction).
Hands-on
-
Haath se RNN: NumPy RNN me
hidden_sizebadal kar parameters ginno aur formula se milao. -
Gates dekho: LSTM ke NumPy code me sequence badal kar dekho forget gate kab bada hota hai.
-
Parameters:
GRU(64)par input 32 ke parameters nikalo (reset_after wala formula). -
Apna review dataset: 20 apne jumlay (positive/negative) likho aur TF-IDF + Logistic baseline banao, phir Colab me LSTM.
-
Experiment: Keras me
LSTMkoGRUse badlo aur parameters, waqt aur accuracy ka muqabla likho.
KEY TAKEAWAYS
-
Sequence data me tarteeb ahem hai; Bag-of-Words tarteeb kho deta hai.
-
RNN har qadam par input aur pichla hidden state mila kar naya hidden state banata hai; weights har qadam par share hote hain.
-
Saada RNN me gradient lambi sequence me vanishing/exploding hota hai. Clipping exploding me madad deti hai.
-
LSTM: forget, input, output gates aur cell state se lambi yaaddasht. GRU: 2 gates, kam parameters.
-
Parameters: SimpleRNN (in+u+1)u, LSTM 4x uske, GRU 3 x (inu + uu + 2u) Keras me.
-
Text pipeline: Tokenization, Padding, Embedding. mask_zero=True se padding nazar-andaz.
-
return_sequences=True agli recurrent layer ke liye; Bidirectional double parameters leta hai.
-
Hamesha pehle saada baseline (TF-IDF + Logistic) banao.