مین مواد پر جائیں

Chapter 22: Advanced Neural Networks aur Optimization Techniques

MODULE 5 | WEEK 22​

Chapter 22: Advanced Neural Networks aur Optimization Techniques​

Pichle chapter me tumne ek chhota network banaya. Asli masail me network gehre hote hain aur unhe train karna ek fun hai: loss ruk jata hai, network overfit kar jata hai, ya training bohat dheemi hoti hai. Is chapter me hum seekhenge ke network ko behtar kaise banayein : gehre networks, overfitting se bachao ( L2, Dropout, Early Stopping, Batch Normalization ), sahi weight initialization , aur optimizers ( SGD, Momentum, RMSprop, Adam ) jo training ko tez aur stable banate hain.

22.1 Gehre networks aur multiclass classification​

Is chapter me ek asli dataset istemal karenge: Digits (Chapter 8 me dekha tha). 1797 haath se likhe hue number (0 se 9) ki 8x8 tasveerein , yani har tasveer ke 64 pixels (0 se 16 tak). Ye multiclass classification hai: 10 classes, to aakhri layer me 10 neurons aur softmax .

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
digits = load_digits()
X, y = digits.data, digits.target
print("X shape:", X.shape, " classes:", np.unique(y))
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
random_state=42, stratify=y)
scaler = StandardScaler().fit(X_tr) # sirf train par fit (leakage nahi)
X_tr_s, X_te_s = scaler.transform(X_tr), scaler.transform(X_te)
print("Train:", X_tr_s.shape, " Test:", X_te_s.shape)
OUTPUT​
X shape: (1797, 64) classes: [0 1 2 3 4 5 6 7 8 9]
Train: (1347, 64) Test: (450, 64)

Ab alag alag gehrai (depth) ke networks aazmate hain. sklearn ka MLPClassifier bhi asli neural network hai (backprop + gradient descent), to ye natije asli hain:

from sklearn.neural_network import MLPClassifier
for layers in [(), (16,), (64,), (128, 64), (128, 64, 32)]:
if layers == ():
from sklearn.linear_model import LogisticRegression
m = LogisticRegression(max_iter=2000)
name = "Koi hidden layer nahi (Logistic)"
else:
m = MLPClassifier(hidden_layer_sizes=layers, max_iter=500, random_state=0)
name = "Hidden layers " + str(layers)
m.fit(X_tr_s, y_tr)
print(name.ljust(34), "test accuracy:", round(m.score(X_te_s, y_te), 3))
OUTPUT​
Koi hidden layer nahi (Logistic) test accuracy: 0.978
Hidden layers (16,) test accuracy: 0.958
Hidden layers (64,) test accuracy: 0.978
Hidden layers (128, 64) test accuracy: 0.98
Hidden layers (128, 64, 32) test accuracy: 0.976

22.2 Overfitting neural network me​

Neural network ke parameters bohat zyada hote hain, is liye wo training data yaad kar leta hai. Pehchanne ka tareeqa: har epoch par training loss aur validation loss dono dekho. Agar training loss girta rahe magar validation loss uper jane lage , to overfitting shuru ho gayi.

OUTPUT​
Validation loss sab se kam epoch: 14
Us waqt train loss: 0.0103 val loss: 0.1445
Aakhri epoch par train loss: 0.0008 val loss: 0.1595
OUTPUT (chart)​
ANALOGY / MISAAL: Learningcurve:trainaurvalidationloss 1.2 Training loss Validationloss 1.0 0.8 loss 0.6 0.4 0.2 0.0 20 40 60 80 100 120 epoch
ANALOGY / MISAAL: Learningcurve:trainaurvalidationloss 1.2 Training loss Validationloss 1.0 0.8 loss 0.6 0.4 0.2 0.0 20 40 60 80 100 120 epoch

22.3 Regularization: overfitting ka ilaaj​

1) L2 Regularization (Weight Decay)​

Chapter 13 me Ridge yaad karo: loss me weights ke squares ka jurmana jodna. Neural network me bhi wohi: bade weights ko saza milti hai, to network saada rehta hai. sklearn me parameter alpha , Keras me kernel_regularizer=regularizers.l2(0.001).

2) Dropout​

Training ke dauran har step par kuch neurons ko random band (0) kar diya jata hai (maslan 30% ya 50%). Network kisi ek neuron par bharosa nahi kar sakta, is liye har neuron mazboot features seekhta hai. Testing me dropout band hota hai.

3) Early Stopping​

Jab validation loss behtar hona band ho jaye (kuch epochs tak), training rok do aur behtareen weights wapas le lo. Ye sab se saada aur asardaar tareeqa hai.

es = MLPClassifier(hidden_layer_sizes=(128, 64), early_stopping=True,
validation_fraction=0.15, n_iter_no_change=10,
max_iter=500, random_state=0)
es.fit(X_tr_s, y_tr)
print("max_iter 500 tha, training ruki:", es.n_iter_, "epochs par")
print("Behtareen validation score:", round(es.best_validation_score_, 3))
print("Test accuracy:", round(es.score(X_te_s, y_te), 3))
OUTPUT​
max_iter 500 tha, training ruki: 31 epochs par
Behtareen validation score: 0.975
Test accuracy: 0.971

4) L2 (alpha) ka asar​

for a in [0.0001, 0.01, 0.1, 1, 3]:
m = MLPClassifier(hidden_layer_sizes=(128, 64), alpha=a,
max_iter=500, random_state=0).fit(X_tr_s, y_tr)
print("alpha =", str(a).ljust(7), "train:", round(m.score(X_tr_s, y_tr), 3),
" test:", round(m.score(X_te_s, y_te), 3))
OUTPUT​
alpha = 0.0001 train:
1.0 test:
0.98
alpha = 0.01 train:
1.0 test:
0.978
alpha = 0.1 train:
1.0 test:
0.984
alpha = 1 train:
1.0 test:
0.98
alpha = 3 train:
0.996 test:
0.976
SUNEHRA USOOL​
  • Pehle Early Stopping lagao, ye hamesha madad karta hai.

  • Phir zaroorat ho to Dropout (0.2 se 0.5) ya L2 aazmao.

  • Sab se asardaar hal aksar zyada data ya chhota network hota hai.

22.4 Batch Normalization​

Gehre networks me har layer ke inputs ka scale badalta rehta hai, jis se training mushkil hoti hai. Batch Normalization har layer ke output ko (har mini-batch par) mean 0 aur std 1 par le aati hai, phir do seekhe hue parameters ( gamma scale ke liye, beta shift ke liye) lagati hai. Nateeja: training tez aur stable .

np.random.seed(0)
batch = np.random.randn(32, 4) * [1, 10, 100, 0.1] + [5, -20, 300, 0] # 4 features, bohat
mukhtalif scale
print("Pehle mean:", batch.mean(axis=0).round(1))
print("Pehle std :", batch.std(axis=0).round(1))
eps = 1e-5
normed = (batch - batch.mean(axis=0)) / np.sqrt(batch.var(axis=0) + eps)
print("Baad me mean:", normed.mean(axis=0).round(2))
print("Baad me std :", normed.std(axis=0).round(2))
OUTPUT​
Pehle mean: [ 5.1 -18.9 297.3 0. ]
Pehle std : [ 1.2 11.1 80.8 0.1]
Baad me mean: [-0. -0. -0. 0.]
Baad me std : [1. 1. 1. 1.]

22.5 Weight Initialization​

Weights ka shuru ka qadar (initial value) bohat ahem hai. Agar sab zero ho to sab neurons ek jaisa seekhte hain (symmetry). Agar bohat bare hon to activations phat jate hain, bohat chhote hon to signal mar jata hai . Do mashhoor tareeqay:

Naam Formula (std) Kis activation ke saath Xavier / Glorot sqrt(2 / (fan_in + fan_out)) Sigmoid, Tanh (Keras ka default) He sqrt(2 / fan_in) ReLU

np.random.seed(0)
def forward_std(init, layers=10, width=256):
a = np.random.randn(1000, width)
stds = []
for _ in range(layers):
fan_in = a.shape[1]
if init == "chhota (0.01)":
W = np.random.randn(fan_in, width) * 0.01
elif init == "bara (1.0)":
W = np.random.randn(fan_in, width) * 1.0
else:
W = np.random.randn(fan_in, width) * np.sqrt(2 / fan_in) # He
a = np.maximum(0, a @ W) # ReLU stds.append(a.std()) return stds
for init in ["chhota (0.01)", "bara (1.0)", "He"]:
s = forward_std(init)
print(init.ljust(14), "layer 1:", format(s[0], ".4g").ljust(10), "layer 5:", format(s[4], ".4g").ljust(11), "layer 10:", format(s[9], ".4g"))

OUTPUT chhota (0.01) layer 1: 0.09312 layer 5: 1.614e-05 layer 10: 3.094e-10 bara (1.0) layer 1: 9.325 layer 5: 1.601e+05 layer 10: 2.532e+10 He layer 1: 0.8284 layer 5: 0.7855 layer 10: 0.6848

He initialization me signal ka size har layer par qareeb qareeb ek jaisa rehta hai, jab ke ghalat scale par ya to ghat kar khatam hota hai ya phat jata hai.

22.6 Vanishing aur Exploding Gradients​

Backprop me gradient har layer par guna hota jata hai. Agar har factor 1 se chhota ho to gradient peeche jate jate khatam ho jata hai ( vanishing ), aur agar bara ho to phat jata hai ( exploding ). Sigmoid ka derivative ziyada se ziyada 0.25 hota hai, to gehre sigmoid networks me pehli layers kuch seekh hi nahi pati.

sig = lambda z:
1 / (1 + np.exp(-z))
np.random.seed(1)
depth = 10
for name, act, dact in [
("Sigmoid", sig, lambda a: a * (1 - a)),
("ReLU", lambda z: np.maximum(0, z), lambda a: (a > 0).astype(float)),
]:
h = np.random.randn(64, 100)
Ws, As = [], [h]
for _ in range(depth):
W = np.random.randn(100, 100) * np.sqrt(2 / 100)
h = act(h @ W)
Ws.append(W)
As.append(h)
g = np.ones_like(h) # aakhri layer par
gradient = 1
mags = []
for W, a in zip(reversed(Ws), reversed(As[1:])):
g = (g * dact(a)) @ W.T
mags.append(np.abs(g).mean())
mags = mags[::-1]
print(name.ljust(8), "gradient size, pehli layer:", format(mags[0], ".2e"),
"| aakhri layer:", format(mags[-1], ".2e"))
OUTPUT​
Sigmoid gradient size, pehli layer: 6.70e-06 | aakhri layer: 2.54e-01
ReLU gradient size, pehli layer: 1.86e+00 | aakhri layer: 8.29e-01

22.7 Optimizers: network ko tez aur behtar seekhana​

Gradient descent weights ko

w = w - lr * gradient

se update karta hai. Isi ke kai behtar roop hain:

OptimizerIdeaFayda / Nuqsan
SGDHar mini-batch ke gradient par seedha qadamSaada; dheema ho sakta hai, learning rate par
bohat nirbhar
SGD + MomentumPichle qadmon ki "raftar" bhi jama rakhoGhatiyon (valleys) me tez, oscillation kam
RMSpropHar parameter ka learning rate gradient ke size
se adjust
Alag alag scale wale parameters ke liye acha
AdamMomentum + RMSprop ka milapSab se mashhoor default; aksar bina tuning acha
chalta hai
OUTPUT​
SGD loss 60 qadam baad: 0.0365 | loss<0.01 hone me qadam: None
Momentum loss 60 qadam baad: 0.0167 | loss<0.01 hone me qadam: 57
RMSprop loss 60 qadam baad: 0.0005 | loss<0.01 hone me qadam: 15
Adam loss 60 qadam baad: 0.0018 | loss<0.01 hone me qadam: 58
xx, yy = np.meshgrid(np.linspace(-8, 4, 200), np.linspace(-3, 3, 200))
zz = 0.1 * xx**2 + 2 * yy**2
cols = {"SGD": "#6B5B95", "Momentum": "#A4457A", "RMSprop": "#2F7D4F", "Adam": "#B7791F"}
fig, ax = plt.subplots(figsize=(9, 4))
ax.contour(xx, yy, zz, levels=18, cmap="Greys", alpha=0.5)
for name, pa in paths.items():
ax.plot(pa[:, 0], pa[:, 1], "o-", ms=3, lw=1.5, color=cols[name], label=name)
ax.plot(0, 0, "k*", ms=14)
ax.set_title("Optimizers ke raaste (ghati ke beech tak)")
ax.legend()
plt.show()
OUTPUT (chart)​
ANALOGY / MISAAL: Optimizerskeraaste(ghatikebeechtak) SGD Momentum RMSprop Adam -1 -2 -3 -6 -4 -2 -8
ANALOGY / MISAAL: Optimizerskeraaste(ghatikebeechtak) SGD Momentum RMSprop Adam -1 -2 -3 -6 -4 -2 -8

Is tajurbe se ek sabaq: koi optimizer har masle me sab se behtar nahi hota, har ek ka asar learning rate aur masle ki shakal par nirbhar hai. Practice me Adam (lr = 0.001) se shuru karna aam hai.

sklearn me solvers ka muqabla (Digits par)

import time
for solver, kw in [("sgd", {"learning_rate_init": 0.01, "momentum": 0}),
("sgd", {"learning_rate_init": 0.01, "momentum": 0.9,
"nesterovs_momentum": True}),
("adam", {"learning_rate_init": 0.001})]:
m = MLPClassifier(hidden_layer_sizes=(128, 64), solver=solver, max_iter=60,
random_state=0, **kw)
import warnings
warnings.filterwarnings("ignore")
m.fit(X_tr_s, y_tr)
tag = solver + (" + momentum" if kw.get("momentum", 0) > 0 else " (plain)")
if solver ==
"sgd"
else "adam"
print(tag.ljust(15), "final train loss:", round(m.loss_, 4),
" test acc:", round(m.score(X_te_s, y_te), 3))
OUTPUT​
sgd (plain) final train loss: 0.4275 test acc: 0.904
sgd + momentum final train loss: 0.0206 test acc: 0.971
adam final train loss: 0.0042 test acc: 0.978

22.8 Learning Rate aur Scheduling​

Learning rate (lr) sab se ahem hyperparameter hai. Bohat bara: loss uchhalta hai ya phat jata hai. Bohat chhota: training bohat dheemi. Scheduling me shuru me bara lr rakhte hain (tez safar) aur baad me chhota karte jate hain (nazuk adjustment).

epochs = np.arange(0, 50)
lr0 = 0.1
step = lr0 * 0.5 ** (epochs // 10) # har 10 epoch par aadha
expo = lr0 * np.exp(-0.06 * epochs) # silsila war ghatna
cosine = 0.5 * lr0 * (1 + np.cos(np.pi * epochs / 50))
print("epoch 0 :", lr0, round(expo[0], 4), round(cosine[0], 4))
print("epoch 25 : step", round(step[25], 4), " exp", round(expo[25], 4), " cosine",
round(cosine[25], 4))
print("epoch 49 : step", round(step[49], 4), " exp", round(expo[49], 4), " cosine",
round(cosine[49], 4))
fig, ax = plt.subplots(figsize=(9, 3.4))
ax.plot(epochs, step, color="#6B5B95", lw=2, label="Step decay")
ax.plot(epochs, expo, color="#A4457A", lw=2, label="Exponential decay")
ax.plot(epochs, cosine, color="#2F7D4F", lw=2, label="Cosine decay")
ax.set_xlabel("epoch")
ax.set_ylabel("learning rate")
ax.legend()
ax.set_title("Learning rate schedules")
plt.show()
OUTPUT​
epoch 0 : 0.1 0.1 0.1
epoch 25 : step 0.025 exp 0.0223 cosine 0.05
epoch 49 : step 0.0062 exp 0.0053 cosine 0.0001
OUTPUT (chart)​
22.8 Learning Rate aur Scheduling: Learning rate schedules 0.10 Step decay Exponentialdecay 0.08 Cosine decay learningrate 0.06 0.04 0.02 0.00 10 20 30 40 50 epoch
22.8 Learning Rate aur Scheduling: Learning rate schedules 0.10 Step decay Exponentialdecay 0.08 Cosine decay learningrate 0.06 0.04 0.02 0.00 10 20 30 40 50 epoch

22.9 Keras me ye sab: gehra network aur callbacks​

Ab sab kuch ek Keras model me jodte hain: Dense + BatchNormalization + Dropout + L2 , Adam optimizer, aur callbacks ( EarlyStopping , ReduceLROnPlateau ). Multiclass labels (0 se 9 integers) hain, to loss sparse_categorical_crossentropy.

from tensorflow import keras
from tensorflow.keras import layers, regularizers, callbacks
model = keras.Sequential([
keras.Input(shape=(64,)),
layers.Dense(128, activation="relu", kernel_regularizer=regularizers.l2(1e-4)),
layers.BatchNormalization(),
layers.Dropout(0.3),
layers.Dense(64, activation="relu"),
layers.Dense(10, activation="softmax"),
])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
model.summary()
OUTPUT (namoona)​
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Layer (type)
┃
┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┳━━━━━━━━━━━━━━━━━━━━━━━━
Output Shape

━━━━━━━━━━━━━━━━━━━━━━━━
━━┳━━━━━━━━━━━━━━━━━┓
Param #
┃
┃
━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense) │
(None, 128)
│ 8,320 │
│ batch_normalization (BatchNormalization) │(None, 128)│ 512 │
│ dropout (Dropout) │(None, 128)│ 0 │
│ dense_1 (Dense) │(None, 64)│ 8,256 │
│ dense_2 (Dense) │
└──────────────────────────────────────────┴
(None, 10)
────────────────────────
│ 650 │
──┴─────────────────┘
Total params: 17,738 (69.29 KB)
Trainable params: 17,482 (68.29 KB)
Non-trainable params: 256 (1.00 KB)
OUTPUT (namoona)​
Training ruki: 31 epochs par (max 200)
Test accuracy: 0.978

22.10 Aam Galtiyan aur Hands-on​

AAM GALTIYAN​
  • Test set par early stopping/tuning karna. Hamesha alag validation set (ya validation_split) use karo.

  • Dropout testing me bhi lagana. Keras predict me ye khud band karta hai, magar apne hath ke code me dhyan do.

  • BatchNorm aur Dropout ko saath saath ghalat tarteeb se lagana; aam tarteeb: Dense, BatchNorm, Activation, Dropout.

  • Adam ke saath bohat bara lr (jaise 0.1) rakhna.

  • Har masle me bara network lagana. Pehle chhota network aur baseline.

  • Data scale na karna aur phir kehna ke "network seekh nahi raha".

Hands-on​

  • Overfit karo: Digits par (256,128) ka network sgd se chalao aur learning curve me val loss ka uper jana khud dekho.

  • Alpha badlo: alpha ko 0.0001 se 3 tak badal kar train aur test accuracy ka farq likho.

  • Optimizer race: upar ke optimizer code me lr badlo (SGD 0.5 par kya hota hai?).

  • Dropout rate: Keras model me dropout 0.1, 0.3, 0.6 aazmao.

  • Callbacks: patience 3 aur 20 rakh kar len(history.history["loss"]) ka muqabla karo.

KEY TAKEAWAYS​
  • Gehra network = zyada taqat magar zyada overfitting ka khatra; chhote se shuru karo.

  • Overfitting ki pehchan: train loss girta rahe, validation loss uper jaye.

  • Regularization: Early Stopping (pehli pasand), L2, Dropout; training me dropout, testing me nahi.

  • Batch Normalization har layer ke inputs ko normalize kar ke training tez aur stable banata hai.

  • Initialization: ReLU ke liye He , sigmoid/tanh ke liye Xavier . Zero initialization kabhi nahi.

  • Vanishing gradient: sigmoid ka derivative <= 0.25; ReLU aur BatchNorm madadgar hain.

  • Optimizers: SGD, Momentum, RMSprop, Adam. Adam (lr=0.001) aam default hai.

  • Learning rate sab se ahem hyperparameter hai; scheduling aur ReduceLROnPlateau use karo.