مین مواد پر جائیں

Chapter 41: Big Data

MODULE 7 | WEEK 41​

Chapter 41: Big Data​

Ab tak tumne jo data dekha wo tumhare laptop ki RAM me aa jata tha (Excel sheet, CSV, Pandas DataFrame). Magar asli duniya me Daraz, Careem, Jazz, banks aur YouTube jaisi companies roz TB (terabyte) aur PB (petabyte) data banati hain, jo ek computer me samata hi nahi. Is chapter me hum seekhenge ke Big Data kya hai , ek computer kyun kam parta hai, aur data ko kai computers par bant kar kaise sambhala jata hai ( HDFS, MapReduce, partitioning, batch aur streaming ). Pehle har idea chhote Python code se hath se samjhenge, phir Chapter 42 me Apache Spark me wohi kaam asaani se karenge.

41.1 Big Data kya hai?​

Big Data us data ko kehte hain jo itna bara, itni tezi se aane wala ya itna mukhtalif ho ke ek computer aur purane tools (jaise Excel ya ek database) use sambhal na sakein. Sirf "bara" hona kaafi nahi; data ki 5 khasusiyaat dekhi jati hain, jinhein 5 V kehte hain.

41.1 Big Data kya hai?: BigDatake5V Volume Velocity Variety Veracity Value bohat zyada data ki raftar data par kaiqism:table, data se text,image,log miqdar (TB/PB) (harsecond) bharosa/saafi faida/insight
41.1 Big Data kya hai?: BigDatake5V Volume Velocity Variety Veracity Value bohat zyada data ki raftar data par kaiqism:table, data se text,image,log miqdar (TB/PB) (harsecond) bharosa/saafi faida/insight
VMatlabMisaal
VolumeData ki miqdar (GB se TB, PB)Ek bank ke saalon ke transactions; YouTube par roz upload hone
wali videos
VelocityData kitni tezi se aa raha haiHar second hazaron UPI/Easypaisa transactions, Twitter/X posts
VarietyData ki kai shakleinTable (structured), JSON/log (semi-structured),
image/video/audio (unstructured)
VeracityData par kitna bharosa hai (galat, adhura,
duplicate)
Sensor ki ghalat reading, ek customer ke do record
ValueData se kaam ki baat nikalnaFraud pakadna, sahi product dikhana, traffic ka andaza

41.2 Ek computer kyun kam parta hai? Scale up vs Scale out​

Jab data barhta hai to do raaste hote hain:

Scale Up (Vertical)Scale Out (Horizontal)
KaiseEk hi computer ko taqatwar banao (zyada RAM, CPU,
bara disk)
Bohat se aam computers ko jod do (cluster)
HadEk had ke baad mehnga aur namumkinZaroorat par aur machines jodte jao
KharabiEk machine kharab = sab kuch bandEk machine kharab = baqi kaam karte rahte hain
Misaal16 GB laptop se 256 GB server100 sasti machines ka cluster
Big Data meKam istemalYehi standard hai(Hadoop, Spark)

Ek chhota hisaab dekhte hain: 1 TB data ek disk se parhne me kitna waqt lagega agar disk 200 MB/second ki raftar se parhe, aur agar wohi data 10 ya 100 disks/machines par bata ho?

size_tb = 1
mb_per_sec = 200 # ek disk ki raftar
total_mb = size_tb * 1024 * 1024
for disks in [1, 10, 100]:
sec = total_mb / (mb_per_sec * disks)
print(f"{disks:>3} disk: {sec:>7.0f} second = {sec/60:>5.1f} minute")
OUTPUT​
1 disk: 5243 second = 87.4 minute
10 disk: 524 second = 8.7 minute
100 disk: 52 second = 0.9 minute

Ek machine par 1 TB parhne me takreeban 87 minute , jab ke 100 machines ek sath parhein to 1 minute se bhi kam . Yehi parallel (ek sath) kaam Big Data ki jaan hai.

41.3 Distributed Storage: HDFS​

HDFS (Hadoop Distributed File System) ek aisa file system hai jo ek bari file ko tukron (blocks) me tor kar kai machines par rakhta hai . Iske teen bunyadi idea hain:

  • Blocks: file ko barabar tukron me torna (aam tor par 128 MB ka block).

  • Replication: har block ki 3 copies alag alag machines par (default). Ek machine kharab ho jaye to data zaya nahi hota.

  • NameNode aur DataNodes: NameNode "fehrist" rakhta hai (kaunsa block kis machine par hai), DataNodes asal blocks rakhte hain.

41.3 Distributed Storage: HDFS: HDFS:file4blocksmetuti,harblock3jagahcopy Node 1 Node 2 Node 3 Node 4 B1 (copy) B1 (copy) B1 (copy) B2 (copy) B3 (copy) B2 (copy) B2 (copy) B3 (copy) B4 (copy) B4 (copy) B3...
41.3 Distributed Storage: HDFS: HDFS:file4blocksmetuti,harblock3jagahcopy Node 1 Node 2 Node 3 Node 4 B1 (copy) B1 (copy) B1 (copy) B2 (copy) B3 (copy) B2 (copy) B2 (copy) B3 (copy) B4 (copy) B4 (copy) B3...
OUTPUT​
Blocks ki tadaad: 4
B1 -> ['N2', 'N5', 'N4']
B2 -> ['N5', 'N4', 'N3']
B3 -> ['N5', 'N1', 'N3']
B4 -> ['N1', 'N4', 'N2']
N2 kharab: True
Do machines ek sath kharab: 10/10 surtoon me file safe

Chunke har block ki 3 copies hain, isliye koi bhi 2 machines ek sath kharab hon to bhi har block ki kam az kam ek copy bachti hai. Ye fault tolerance (kharabi bardasht karna) kehlata hai.

41.4 MapReduce: kaam ko bant kar karne ka tareeqa​

MapReduce ek programming model hai jo kisi bhi bare kaam ko 3 qadam me bantta hai:

  • Map: har machine apne hisse ke data par kaam kare aur (key, value) jode banaye.

  • Shuffle: wo jode jinki key ek jaisi hai unhein ek jagah jama kiya jaye.

  • Reduce: har key ke tamam values ko jod kar (sum, count, max) ek jawab banaya jaye.

41.4 MapReduce: kaam ko bant kar karne ka tareeqa: 1.Map 3.Reduce Map1 2.Shuffle Reduce 1 Input Shuffle (key ke hisab Out file Map2 se jama) put Reduce2 Map3
41.4 MapReduce: kaam ko bant kar karne ka tareeqa: 1.Map 3.Reduce Map1 2.Shuffle Reduce 1 Input Shuffle (key ke hisab Out file Map2 se jama) put Reduce2 Map3

Word Count: MapReduce ka "Hello World"​

from collections import defaultdict
lines = ["spark tez hai",
"hadoop purana hai spark naya hai",
"big data hai"]
# 1) MAP:
har lafz ke liye (lafz, 1)
mapped = []
for line in lines:
for word in line.split():
mapped.append((word, 1))
print("MAP :", mapped[:4], "...")
# 2) SHUFFLE: ek jaisi key ek jagah
groups = defaultdict(list)
for key, value in mapped:
groups[key].append(value)
print("SHUFFLE:")
for key, vals in groups.items():
print(" ", key, vals)
# 3) REDUCE: har key ke values ka jama
result = {key: sum(vals) for key, vals in groups.items()}
print("REDUCE :")
for key, total in result.items():
print(" ", key, total)

OUTPUT

MAP : [('spark', 1), ('tez', 1), ('hai', 1), ('hadoop', 1)] ...
SHUFFLE:
spark [1, 1]
tez [1]
hai [1, 1, 1, 1]
hadoop [1]
purana [1]
naya [1]
big [1]
data [1]
REDUCE :
spark 2
tez 1
hai 4
hadoop 1
purana 1
naya 1
big 1
data 1

Asli Hadoop me Map aur Reduce alag alag machines par chalte hain aur beech ka data disk par likha jata hai, jis wajah se MapReduce sust hota hai. Isi kamzori ko door karne ke liye Spark aaya (Chapter 42).

41.5 Data ke formats: Row-based vs Columnar​

Big Data me data ko kis shakal me rakha jaye ye bhi bohat ahem hai. Ek file ko do tareeqon se rakha ja sakta hai:

  • Row-based (CSV, JSON): ek row ke sare columns ek sath. Ek record dhoondne ya likhne ke liye acha.

  • Columnar (Parquet, ORC): ek column ki sari values ek sath. Analytics (sirf kuch columns ka sum/average) ke liye bohat tez aur chhota.

Maslan 1000 rows me se sirf amount column ka jama chahiye. Dekhte hain dono tareeqon me kitni values chhoo kar guzarni parti hain:

# row-store: har row = (id, city, amount, qty)
rows = [(i, "Karachi", 100 + i, 5) for i in range(1000)]
columns = { # column-store
"id": [r[0] for r in rows],
"city": [r[1] for r in rows],
"amount": [r[2] for r in rows],
"qty": [r[3] for r in rows],
}
# Row-store: har row poori parhni parti hai (4 values)
touched_row = sum(len(r) for r in rows)
total_row = sum(r[2] for r in rows)
# Column-store: sirf 'amount' column
touched_col = len(columns["amount"])
total_col = sum(columns["amount"])
print("Row-store :", touched_row, "values chhui, jama =", total_row)
print("Column-store:", touched_col, "values chhui, jama =", total_col)
OUTPUT​
Row-store : 4000 values chhui, jama = 599500
Column-store: 1000 values chhui, jama = 599500
FormatQismKhaas baatKab istemal karein
CSVRow, textSaada, har jagah khulta hai; bara aur sustChhota data, sharing
JSONRow, textNested data rakh sakta hai; baraAPIs, logs
ParquetColumnar, binaryCompressed, schema sath, sirf zaroori columns
parhta hai
Big Data analytics ka standard
ORCColumnar, binaryParquet jaisa, Hive me aamHive/Hadoop ecosystem

41.6 Partitioning aur Data Skew​

Data ko machines me bantne ka kaam partitioning kehlata hai. Sab se aam tareeqa hash partitioning hai: har record ki key ka hash nikal kar hash % partitions se tay hota hai ke wo kis partition me jayega. Achi partitioning me har partition ke paas taqreeban barabar kaam hota hai.

def h(key):
# saada deterministic hash
v = 0
for ch in str(key):
v = (v * 31 + ord(ch)) % 1000003
return v
def partition_sizes(keys, n=4):
sizes = [0] * n
for k in keys:
sizes[h(k) % n] += 1
return sizes
balanced = [f"cust{i}" for i in range(1000)]
skewed = ["cust_hot"] * 600 + [f"cust{i}" for i in range(400)]
print("Balanced:", partition_sizes(balanced))
print("Skewed :", partition_sizes(skewed))
OUTPUT​
Balanced: [251, 249, 249, 251]
Skewed : [102, 700, 98, 100]
41.6 Partitioning aur Data Skew: Achipartitioning (balanced) Skew:ekkeybohatbhari 700 700 600 rows ki tadaad 500 400 300 251 249 249 251 200 102 98 100 100 PO P1 P2 P3 PO P1 P2 P3
41.6 Partitioning aur Data Skew: Achipartitioning (balanced) Skew:ekkeybohatbhari 700 700 600 rows ki tadaad 500 400 300 251 249 249 251 200 102 98 100 100 PO P1 P2 P3 PO P1 P2 P3

Doosri soorat me ek key ( cust_hot ) ke 600 records hain, to ek partition par sara bojh aa gaya aur baqi machines farigh baithi hain. Ye Data Skew hai: poora job sab se sust partition ke barabar sust ho jata hai. Hal: behtar key chunna, salting (key me random lafz jorna) ya skew wali key ko alag sambhalna.

41.7 Batch vs Streaming​

BatchStreaming
DataJama shuda, poori file/tableMusalsal aata hua (event by event)
Kab chalta haiWaqt tay (har raat, har ghanta)Har waqt, chand second ke andar
LatencyMinute se ghanteMillisecond se second
MisaalRaat ko din bhar ki sales ka reportCard fraud ka foran alert, live traffic
ToolsHadoop MapReduce, SparkKafka, Spark Structured Streaming, Flink

Streaming me data ko aksar time window me bant kar ginte hain. Neeche 5 second ke tumbling windows ki chhoti si misaal hai:

# har event = (second, user)
events = [(1, "A"), (3, "B"), (4, "A"), (7, "A"), (8, "B"), (11, "A")]
window = 5
counts = {}
for t, user in events:
start = (t // window) * window
counts[start] = counts.get(start, 0) + 1
for start in sorted(counts):
print(f"[{start:>2}s se {start + window:>2}s): {counts[start]} events")
OUTPUT​
[ 0s se 5s): 3 events
[ 5s se 10s): 2 events
[10s se 15s): 1 events

41.8 Big Data Ecosystem ek nazar me​

KaamToolEk line me
StorageHDFS, S3, Azure Blob/ADLSBara data kai machines/cloud par rakhna
KaamToolEk line me
Batch processingHadoop MapReduce,Apache SparkBare data par hisab kitab; Spark zyada tez
SQL on big dataHive, Spark SQL, Presto/Trino, AthenaBare data par SQL chalana
Messaging / ingestApache Kafka, Kinesis, Event HubsLive data ko pipeline me laana
Stream processingSpark Structured Streaming,FlinkLive data par hisab
NoSQL databaseMongoDB, Cassandra, HBase, DynamoDBBohat tez likhna/parhna, lachakdaar schema
OrchestrationAirflow, Azure Data FactoryKaam ka silsila aur schedule

41.9 Data Lake vs Data Warehouse​

Data WarehouseData Lake
Data ki shakalSaaf, structured tablesHar qism ka raw data (CSV, JSON, image, log)
SchemaPehle tay (schema-on-write)Parhte waqt tay (schema-on-read)
QeematZyadaKam (sasta object storage)
Kis ke liyeBI dashboards, reportsData science, ML, raw archive
MisaalRedshift, Synapse, BigQuery, SnowflakeS3, Azure Data Lake Storage

Aaj ka rujhan Lakehouse hai: Data Lake ke sasta storage par Warehouse jaisi tables aur SQL (Delta Lake, Apache Iceberg). Is par Chapter 43-44 me bhi baat hogi.

41.10 Aam Galtiyan aur Hands-on​

AAM GALTIYAN​
  • Har bare lagne wale data ko Big Data keh dena. Agar data aapke laptop me aa jata hai aur Pandas theek chal raha hai, to Pandas hi behtar hai.

  • Sirf Volume ko Big Data samajhna; Velocity aur Variety bhi utni hi ahem hain.

  • CSV ko hi har jagah rakhna ; analytics ke liye Parquet kai guna tez hai.

  • Partition key sochay bina chunna (skew ka masla).

  • Sochna ke replication backup ki jagah le leta hai. Replication kharabi se bachata hai, galti se delete ya corruption se nahi; backup alag zaroori hai.

Hands-on​
  • Scale: 41.2 me
size_tb = 10

aur

disks = 500

rakh kar hisaab lagao; kitna waqt laga?

  • HDFS:
file_mb = 1000

aur

replicas = 2

karo. Kya ab bhi koi 2 machines ek sath kharab hone par file safe rehti hai? Kyun/kyun nahi?

  • MapReduce: lines me apne pasandeeda 5 jumle daalo aur word count nikalo; sab se zyada aane wala lafz kaunsa hai?

  • Columnar: row-store aur column-store me ab qty ka average nikalo aur dekho kitni values chhui gayin.

  • Skew: skewed list me cust_hot ki tadaad 600 se 100 karo aur partition sizes dekho.

  • Streaming:

window = 3

karo aur nateeja dekho.

KEY TAKEAWAYS​
  • Big Data = bohat bara, tez ya mukhtalif data jo ek computer sambhal na sake: 5 V (Volume, Velocity, Variety, Veracity, Value).

  • Scale out (kai sasti machines) Big Data ka standard hal hai; scale up ki had hoti hai.

  • HDFS: file blocks me tutti hai aur har block ki 3 copies hoti hain ( fault tolerance ).

  • MapReduce: Map (key, value banao) -> Shuffle (key ke hisab se jama) -> Reduce (jama/ginti).

  • Analytics ke liye columnar formats (Parquet) row-based formats (CSV) se tez aur chhote hote hain.

  • Partitioning kaam bantti hai; data skew se ek machine par bojh aa jata hai.

  • Batch = jama data, Streaming = musalsal data; Data Lake (raw, sasta) vs Warehouse (saaf, structured).