مین مواد پر جائیں

Chapter 25: Recurrent Neural Networks (RNN), LSTM aur GRU

MODULE 5 | WEEK 25​

Chapter 25: Recurrent Neural Networks (RNN), LSTM aur GRU​

Ab tak ka data (table, tasveer) tarteeb ke baghair tha. Magar zindagi me bohat sa data tarteeb wala (sequence) hota hai: jumlay ke alfaz, awaaz, share ki qeemat, mausam. Is me pehle kya aaya ye baad wale ko samajhne me madad deta hai. Is chapter me RNN (jis me network ki ek yaaddasht hoti hai), uski kamzori ( vanishing gradient ), aur uske behtar roop LSTM aur GRU seekhenge, aur dekhenge ke text ko network me kaise dalte hain ( Embedding ).

25.1 Sequence data kya hai?​

Sequence woh data hai jis me tarteeb ahem hai aur lambai badal sakti hai: jumlay, gaana, mausam ki rozana reading, share ka bhao, DNA. Aam Dense network ke liye ye teen masail paida karta hai:

  • Tarteeb: "kutta insan ko kata" aur "insan ne kutte ko kata" me wohi alfaz hain magar matlab ulta.

  • Lambai alag alag: koi jumla 5 lafz ka, koi 50 ka. Dense layer ko fixed input chahiye.

  • Pichli maloomat: jumle ke aakhir ka matlab shuru ke lafz par nirbhar ho sakta hai ("meri behan jo Lahore me rehti hai, wo aaj aa rahi hai").

Dekhte hain ke Bag-of-Words (sirf alfaz ginna, tarteeb nahi) kaise tarteeb kho deta hai:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
sentences = ["dog bites man", "man bites dog"]
cv = CountVectorizer()
M = cv.fit_transform(sentences).toarray()
print("Vocabulary:", cv.get_feature_names_out())
print(M)
print("Dono jumle ek jaise numbers?", (M[0] == M[1]).all())
OUTPUT​
Vocabulary: ['bites' 'dog' 'man']
[[1 1 1]
[1 1 1]]
Dono jumle ek jaise numbers? True

Do bilkul mukhtalif matlab wale jumle, magar Bag-of-Words ke liye barabar . RNN tarteeb ko bachata hai.

25.2 RNN: network ki yaaddasht​

Recurrent Neural Network sequence ko ek ek qadam (time step) par parhta hai. Har qadam par wo naye input x_t ke saath pichla hidden state h_(t-1) (apni yaaddasht) bhi leta hai aur naya h_t banata hai. Wohi weights har qadam par dobara istemal hote hain.

h_t = tanh( Wx * x_t + Wh * h_(t-1) + b )
x_t :
is qadam ka input
h_(t-1) :
pichli yaaddasht
h_t :
nayi yaaddasht (agle qadam ko jati hai)
OUTPUT​
t = 0
h = [0.707 0.197 0.454 0.808]
t = 1
h = [ 0.934 -0.719 0.705 -0.278]
t = 2
h = [-0.896 0.646 0.434 0.742]
t = 3
h = [ 0.978 -0.668 0.79 0.37 ]
Weights ki shapes:
Wx (3, 4)
Wh (4, 4)
Parameters (kitni bhi lambi sequence ho):
32

25.3 RNN ke amli roop​

QismInput -> OutputMisaal
Many-to-OnePoori sequence -> ek jawabReview se sentiment (positive/negative)
One-to-ManyEk input -> sequenceTasveer se caption
Many-to-Many (barabar)Har qadam par ek outputHar lafz ka part-of-speech
QismInput -> OutputMisaal
Many-to-Many (encoder-decoder)Sequence -> doosri sequenceTranslation (Angrezi -> Urdu)

Training ke liye network ko waqt me khol (unroll) diya jata hai: 4 qadam ki sequence ke liye aisa lagta hai jaise 4 layers ka network hai jis ke weights sab layers me barabar hain. Backpropagation isi unrolled network par chalta hai, isay Backpropagation Through Time (BPTT) kehte hain.

25.4 RNN ki kamzori: Vanishing Gradient​

BPTT me gradient ko har qadam par Wh se guna kar ke peeche bheja jata hai. Agar Wh "chhota" ho to lambi sequence me gradient mar jata hai (network shuru ki baatein bhool jata hai), aur agar bara ho to phat jata hai.

np.random.seed(1)
n = 20
Q, _ = np.linalg.qr(np.random.randn(n, n)) # seedha (orthogonal) matrix
for radius in [0.9, 1.0, 1.1]:
Wh = Q * radius # har ghumao me size 'radius' guna
v = np.ones(n)
sizes = []
for t in range(1, 51):
v = Wh.T @ v # ek qadam peeche
sizes.append(np.linalg.norm(v) / np.sqrt(n)) # shuru ke size ke muqable me
print("Wh ka size", radius, " -> 10 qadam peeche:", format(sizes[9], ".3f"), "guna",
" | 50 qadam peeche:", format(sizes[49], ".3f"), "guna")
OUTPUT​
Wh ka size 0.9 -> 10 qadam peeche: 0.349 guna | 50 qadam peeche: 0.005 guna
Wh ka size 1.0 -> 10 qadam peeche: 1.000 guna | 50 qadam peeche: 1.000 guna
Wh ka size 1.1 -> 10 qadam peeche: 2.594 guna | 50 qadam peeche: 117.391 guna

Jab Wh ka asar 0.9 ho, 50 qadam peeche gradient shuru ke size ka sirf 0.005 guna reh jata hai ( vanishing ); 1.1 par 117 guna barh jata hai ( exploding ). Is liye saada RNN lambi sequence me purani maloomat yaad nahi rakh pata.

25.5 LSTM: lambi yaaddasht​

LSTM (Long Short-Term Memory) me ek alag cell state (C_t) hota hai, jo ek "sadak" ki tarah poori sequence se guzarta hai, aur teen gates (darwaze) tay karte hain ke is sadak par kya likhna, kya mitana aur kya parhna hai:

GateSawal jo wo poochta haiNateeja (0 se 1)
Forget gate (f)Purani yaaddasht me se kitnabhoolun?0 = sab bhool, 1 = sab yaad
Input gate (i)Nayi maloomat me se kitnalikhun?0 = kuch nahi, 1 = poora
Output gate (o)Cell state me se kitnabahar dikhaun?0 = kuch nahi, 1 = poora
f = sigmoid( [x_t, h_(t-1)] @ Wf + bf ) forget gate
i = sigmoid( [x_t, h_(t-1)] @ Wi + bi ) input gate
o = sigmoid( [x_t, h_(t-1)] @ Wo + bo ) output gate
g = tanh ( [x_t, h_(t-1)] @ Wg + bg ) nayi umeedwar maloomat
C_t = f * C_(t-1) + i * g cell state update
h_t = o * tanh(C_t) naya hidden state
OUTPUT​
t = 0 | forget:
[0.45 0.49 0.26 0.69] | input:
[0.33 0.48 0.47 0.75] | output:
[0.42 0.56
0.53 0.28]
t = 1 | forget:
[0.27 0.37 0.57 0.32] | input:
[0.2 0.52 0.56 0.67] | output:
[0.4 0.48
0.54 0.49]
t = 2 | forget:
[0.35 0.39 0.56 0.73] | input:
[0.54 0.39 0.54 0.57] | output:
[0.21 0.5
0.45 0.31]
t = 3 | forget:
[0.25 0.41 0.28 0.54] | input:
[0.14 0.49 0.52 0.87] | output:
[0.38 0.55
0.51 0.38]
Aakhri hidden state h:
[-0.021 -0.078 0.139 0.27 ]
Aakhri cell state C:
[-0.056 -0.144 0.279 0.892]
LSTM parameters:
128

25.6 GRU: halka muqabil​

GRU (Gated Recurrent Unit) LSTM ka saada roop hai: cell state alag nahi , aur sirf do gates : update gate (kitna purana rakhun, kitna naya lun) aur reset gate (pichli yaaddasht ka kitna hissa naya hisaab lagate waqt dekhun). Kam parameters, aksar LSTM jaisi accuracy, aur tez training.

PehluSimpleRNNLSTMGRU
GatesKoi nahi3 (+ cell state)2
Lambi yaaddashtKamzorMazbootMazboot
ParametersSab se kamSab se zyadaBeech me
RaftarSab se tezSab se dheemiLSTM se tez
Aam istemalSeekhne ke liye, chhoti
sequence
Default pasandJab data kam ya raftar chahiye

Parameters ki ginti​

SimpleRNN : (input + units + 1) x units
LSTM : 4 x (input + units + 1) x units
GRU (Keras): 3 x (input x units + units x units + 2 x units) (Keras me do bias set hote
hain)
def rnn_p(inp, u): return (inp + u + 1) * u
def lstm_p(inp, u): return 4 * (inp + u + 1) * u
def gru_p(inp, u): return 3 * (inp * u + u * u + 2 * u)
inp, units = 8, 16
print("SimpleRNN(16), input 8:", rnn_p(inp, units))
print("LSTM(16), input 8:", lstm_p(inp, units))
print("GRU(16), input 8:", gru_p(inp, units))

OUTPUT SimpleRNN(16), input 8: 400 LSTM(16), input 8: 1600 GRU(16), input 8: 1248

25.7 Text ko network me dalna​

Network sirf numbers samajhta hai, to text ko 3 qadmon me badalte hain: Tokenization (alfaz ko ginti de do), Padding (sab jumlon ko barabar lambai ka karo), aur Embedding (har lafz ko seekhe hue numbers ki list (vector) de do).

reviews = [
"khana bohat acha tha",
"service bohat kharab thi",
"acha khana aur acha mahaul",
"bohat kharab khana",
"mahaul acha tha khana lajawab",
"kharab service aur kharab khana",
]
labels = np.array([1, 0, 1, 0, 1, 0]) # 1 = positive, 0 = negative
# 1) Tokenization:
har naye lafz ko ginti (0 padding ke liye)
vocab = {}
for r in reviews:
for w in r.split():
if w not in vocab:
vocab[w] = len(vocab) + 1
print("Vocabulary:", vocab)
encoded = [[vocab[w] for w in r.split()] for r in reviews]
print("Encoded :", encoded)
# 2) Padding:
sab ko 5 lafz ki lambai tak (peeche zero)
max_len = 5
padded = np.array([seq + [0] * (max_len - len(seq)) for seq in encoded])
print("Padded:")
print(padded)
OUTPUT​
Vocabulary: {'khana': 1, 'bohat': 2, 'acha': 3, 'tha': 4, 'service': 5, 'kharab': 6, 'thi':
7, 'aur': 8, 'mahaul': 9, 'lajawab': 10}
Encoded : [[1, 2, 3, 4], [5, 2, 6, 7], [3, 1, 8, 3, 9], [2, 6, 1], [9, 3, 4, 1, 10], [6,
5, 8, 6, 1]]
Padded:
[[ 1 2 3 4 0]
[ 5 2 6 7 0]
[ 3 1 8 3 9]
[ 2 6 1 0 0]
[ 9 3 4 1 10]
[ 6 5 8 6 1]]

Embedding layer ek seekhne wali table hai: shape (vocab_size, embedding_dim). Har lafz ki ginti us table ki ek row chunti hai, aur ye rows training me seekhi jati hain (milte julte matlab wale alfaz qareeb aate jate hain). Ye one-hot encoding se bohat behtar hai, kyun ke chhoti aur maani se bhari hoti hai.

vocab_size, emb_dim = len(vocab) + 1, 3 # +1 padding ke liye
embedding = np.random.RandomState(0).randn(vocab_size, emb_dim) * 0.1 # shuru me random
seq = padded[0] # "khana bohat acha tha" + padding
vectors = embedding[seq] # har lafz ki row
print("Ek jumle ki shape: (lafz, embedding_dim) =", vectors.shape)
print("Embedding table ki shape:", embedding.shape)
print("Embedding ke parameters:", embedding.size)
OUTPUT​
Ek jumle ki shape: (lafz, embedding_dim) = (5, 3)
Embedding table ki shape: (11, 3)
Embedding ke parameters: 33

25.8 Keras me RNN, LSTM, GRU​

from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential([
keras.Input(shape=(20,), dtype="int32"), # 20 lafz ki jumla (ginti ki list)
layers.Embedding(input_dim=1000, output_dim=16, mask_zero=True),
layers.LSTM(32), # many-to-one: aakhri hidden state
layers.Dense(1, activation="sigmoid"), # positive/negative
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.summary()
OUTPUT (namoona)​
Model: "sequential"
Layer (type)Output shapeParam #
embedding (Embedding)(None, 20, 16)16,000
lstm (LSTM)(None, 32)6,272
dense (Dense)(None, 1)33
Total params22,305 (87.13 KB)
Trainable params22,305 (87.13 KB)
Non-trainable params0 (0.00 B)

return_sequences aur layers ko stack karna​

Agar agli recurrent layer ko har qadam ka output chahiye to return_sequences=True lagao. Do tarfa ( Bidirectional ) layer jumle ko aage se aur peeche se dono taraf parhti hai (jab poora jumla pehle se maujood ho, jaise review):

model2 = keras.Sequential([
keras.Input(shape=(20,), dtype="int32"),
layers.Embedding(1000, 16, mask_zero=True),
layers.Bidirectional(layers.GRU(32, return_sequences=True)), # har qadam ka output
layers.Bidirectional(layers.GRU(16)), # aakhri
layers.Dropout(0.3),
layers.Dense(1, activation="sigmoid"),
])
model2.summary()
OUTPUT (namoona)​
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type) Output Shape Param # ┃┃┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ embedding_1 (Embedding) │ (None, 20, 16) │ 16,000 │
│ bidirectional (Bidirectional) │ (None, 20, 64) │ 9,600 │
│ bidirectional_1 (Bidirectional) │ (None, 32) │ 7,872 │
│ dropout (Dropout) │ (None, 32) │ 0 │
│ dense_1 (Dense) │ (None, 1) │ 33 │
└───────────────────────────────────┴──────────────────────────┴─────────────────┘
Total params: 33,505 (130.88 KB)
Trainable params: 33,505 (130.88 KB)
Non-trainable params: 0 (0.00 B)
OUTPUT​
Baseline predictions: [1 0 1] (1 = positive)

Chhote data par baseline bohat acha chal jata hai; RNN/LSTM ka asli faida bare data aur lambe jumlon par hota hai.

25.9 Aam Galtiyan aur Hands-on​

AAM GALTIYAN​
  • Padding aur masking bhoolna: network zero padding ko "lafz" samajh leta hai.

  • Sequences ko sort ya shuffle kar dena (waqt ki tarteeb bigad jati hai, khaaskar time series me).

  • Bohat lambi sequence par saada RNN (LSTM/GRU lagao).

  • Gradient clipping na lagana jab loss achanak nan ho jaye.

  • Test data ka vocabulary training me shamil karna (leakage).

  • Bidirectional wahan lagana jahan agla lafz maloom nahi hota (jaise agle lafz ki prediction).

Hands-on​

  • Haath se RNN: NumPy RNN me hidden_size badal kar parameters ginno aur formula se milao.

  • Gates dekho: LSTM ke NumPy code me sequence badal kar dekho forget gate kab bada hota hai.

  • Parameters: GRU(64) par input 32 ke parameters nikalo (reset_after wala formula).

  • Apna review dataset: 20 apne jumlay (positive/negative) likho aur TF-IDF + Logistic baseline banao, phir Colab me LSTM.

  • Experiment: Keras me LSTM ko GRU se badlo aur parameters, waqt aur accuracy ka muqabla likho.

KEY TAKEAWAYS​
  • Sequence data me tarteeb ahem hai; Bag-of-Words tarteeb kho deta hai.

  • RNN har qadam par input aur pichla hidden state mila kar naya hidden state banata hai; weights har qadam par share hote hain.

  • Saada RNN me gradient lambi sequence me vanishing/exploding hota hai. Clipping exploding me madad deti hai.

  • LSTM: forget, input, output gates aur cell state se lambi yaaddasht. GRU: 2 gates, kam parameters.

  • Parameters: SimpleRNN (in+u+1)u, LSTM 4x uske, GRU 3 x (inu + uu + 2u) Keras me.

  • Text pipeline: Tokenization, Padding, Embedding. mask_zero=True se padding nazar-andaz.

  • return_sequences=True agli recurrent layer ke liye; Bidirectional double parameters leta hai.

  • Hamesha pehle saada baseline (TF-IDF + Logistic) banao.