Skip to main content

Chapter 42: Apache Spark

MODULE 7 | WEEK 42​

Chapter 42: Apache Spark​

In Chapter 41 we saw that work on big data is done by splitting it across many machines , and the old MapReduce writes data to disk after every step, which makes it slow. Apache Spark does the same work in memory (RAM) and offers an easy API in Python, SQL, Scala and Java. Most Big Data companies in the world use Spark today. In this chapter we will learn PySpark (the Python side of Spark): RDD, DataFrame, Spark SQL, lazy evaluation, joins, window functions, performance and Parquet , and finally a short look at MLlib .

TOPIC CHECKLIST (tick them off as you go)​
  • ☐ What Spark is and why it is faster than MapReduce

  • ☐ Spark's structure: Driver, Executors, Cluster Manager

  • ☐ SparkSession and setup in Colab

  • ☐ RDD: parallelize, map, filter, reduceByKey

  • ☐ DataFrame: schema, select, filter, withColumn, groupBy, agg

  • ☐ Spark SQL (temporary view)

  • ☐ Lazy evaluation, Transformations vs Actions, DAG

  • ☐ Joins and Window functions

  • ☐ Performance: partitions, cache, broadcast join, Parquet

  • ☐ MLlib: a small Pipeline example

  • ☐ Pandas vs Spark: which to use when

42.1 What is Spark and why is it fast?​

Apache Spark is an open-source distributed computing engine that does fast, parallel work on big data. It has three big advantages:

  • In-memory processing: intermediate results stay in RAM instead of being written to disk at every step. That is why it is many times faster than MapReduce for iterative work (machine learning).

  • One engine, many jobs: batch, SQL, streaming, machine learning (MLlib), graphs.

  • Easy API: Python (PySpark), SQL, Scala, Java, R.

ComponentJob
Spark Core (RDD)Foundation: splitting work, memory, fault tolerance
Spark SQL / DataFrameTable-like data and SQL; the most widely used
Structured StreamingThe same DataFrame code on live data
MLlibDistributed machine learning
GraphXGraph analysis (Scala/Java)
42.1 What is Spark and why is it fast?: ApacheSparkarchitecture Executor1 Task|Task|Cache Driver Cluster (SparkSession, Manager (YARN/K8s) Executor2 makes theplan) Standalone) TaskTask|Cache Executor3 TaskITask/Cache
42.1 What is Spark and why is it fast?: ApacheSparkarchitecture Executor1 Task|Task|Cache Driver Cluster (SparkSession, Manager (YARN/K8s) Executor2 makes theplan) Standalone) TaskTask|Cache Executor3 TaskITask/Cache
ComponentWhat it does
DriverYour program runs here.SparkSessionlives here. It makes theplanfor the work and sends out
tasks.
Cluster ManagerHands out the machines' resources (CPU/RAM) (Standalone, YARN, Kubernetes).
ExecutorProcesses on worker machines thatrun tasksandcachedata.
Task / PartitionOne piece of data and the small job done on it. One partition = one task.

42.2 Setup: SparkSession and the dataset​

In Colab, install it in one cell and create a SparkSession . local[*] means use all the cores of your one machine (enough for learning; on a cluster this changes).

!pip install pyspark
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.appName("Module8")
.master("local[*]")
.getOrCreate())
print(spark.version)

OUTPUT (sample; your version may differ) 4.0.0

Now we create 5000 rows of sales data for this chapter and Chapter 44. This code runs anywhere (Python only), and because of seed 42 it produces exactly the same data every time:

import random, csv
from datetime import date, timedelta
random.seed(42)
cities = ["Karachi", "Lahore", "Islamabad", "Quetta", "Peshawar"]
price_range = {"Mobile": (25000, 90000), "Laptop": (60000, 250000), "Headphones": (2000, 15000), "Watch": (5000, 40000)}
start = date(2025, 1, 1)
with open("sales.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["order_id", "order_date", "city", "category", "qty", "unit_price", "customer_id"])
for i in range(1, 5001):
cat = random.choice(list(price_range))
lo, hi = price_range[cat]
d = start + timedelta(days=random.randint(0, 364))
w.writerow([i, d.isoformat(), random.choice(cities), cat, random.randint(1, 4), random.randrange(lo, hi, 500), random.randint(1, 800)])
print("sales.csv is ready")

OUTPUT sales.csv is ready

In Colab, run the file and the code above in the same place ( sales.csv will appear in the Files panel). The first 3 rows:

OUTPUT (sample; the first 3 rows of sales.csv)

+--------+----------+---------+--------+---+----------+-----------+
|order_id|order_date| city|category|qty|unit_price|customer_id|
+--------+----------+---------+--------+---+----------+-----------+
| 1|2025-01-13|Islamabad| Mobile| 2| 53500| 143|
| 2|2025-12-13| Peshawar| Mobile| 1| 79000| 33|
| 3|2025-02-17| Lahore| Mobile| 2| 89500| 617|
+--------+----------+---------+--------+---+----------+-----------+

42.3 RDD: the foundation of Spark​

RDD (Resilient Distributed Dataset) is Spark's oldest and most basic data structure: a list that is spread across many machines (partitions) . Resilient means: if a machine fails, Spark rebuilds that part from the remembered plan (lineage) . Nowadays we mostly use DataFrame , but understanding RDD is understanding Spark.

sc = spark.sparkContext
lines = sc.parallelize([
"spark is fast",
"hadoop is old spark is new",
"big data is",
])
counts = (lines
.flatMap(lambda line: line.split()) # each sentence -> words
.map(lambda w: (w, 1)) # (word, 1)
.reduceByKey(lambda a, b: a + b)) # sum by key
print(counts.sortBy(lambda kv: (-kv[1], kv[0])).collect())

OUTPUT (sample) [('is', 4), ('spark', 2), ('big', 1), ('data', 1), ('fast', 1), ('hadoop', 1), ('new', 1), ('old', 1)]

This is the same Word Count we did by hand in Chapter 41; now flatMap / map is the Map and reduceByKey is the Shuffle + Reduce . The only difference is that Spark spreads this work across many machines by itself.

OperationTypeWhat it does
map(f)TransformationApplies f to every element
filter(f)TransformationKeeps only the elements for which f is True
flatMap(f)TransformationProduces 0 or more elements from each element
reduceByKey(f)Transformation (shuffle)Combines the values of the same key
collect()ActionBrings all the data to the driver (dangerous on big data)
count(),take(n)ActionCount / the first n elements

42.4 DataFrame: table-like distributed data​

DataFrame looks like a Pandas DataFrame (rows and named columns), but underneath it is spread across many machines and Spark's optimizer (Catalyst) makes the work on it faster. That is why DataFrame/SQL is the first choice in production.

Reading a CSV (write the schema yourself)​

inferSchema=True reads the data twice (slow), and may also get a type wrong. For big data it is better to write the schema yourself :

from pyspark.sql.types import (
StructType,
StructField,
IntegerType,
StringType,
DateType,
)

schema = StructType([
StructField("order_id", IntegerType(), True),
StructField("order_date", DateType(), True),
StructField("city", StringType(), True),
StructField("category", StringType(), True),
StructField("qty", IntegerType(), True),
StructField("unit_price", IntegerType(), True),
StructField("customer_id", IntegerType(), True),
])
sales = spark.read.csv("sales.csv", header=True, schema=schema)
sales.printSchema()
print(sales.count())
OUTPUT (sample)​
root
|-- order_id: integer (nullable = true)
|-- order_date: date (nullable = true)
|-- city: string (nullable = true)
|-- category: string (nullable = true)
|-- qty: integer (nullable = true)
|-- unit_price: integer (nullable = true)
|-- customer_id: integer (nullable = true)
5000

select, filter, withColumn​

from pyspark.sql import functions as F
big = (sales
.withColumn("revenue", F.col("qty") * F.col("unit_price")) # new column
.filter(F.col("city") == "Karachi") # Karachi only
.select("order_id", "category", "qty", "unit_price", "revenue"))
big.show(3)

OUTPUT (sample)

+--------+--------+---+----------+-------+
|order_id|category|qty|unit_price|revenue|
+--------+--------+---+----------+-------+
| 8| Watch| 4| 10000| 40000|
| 15| Mobile| 3| 76000| 228000|
| 20| Mobile| 2| 45000| 90000|
+--------+--------+---+----------+-------+

groupBy and agg​

sales = sales.withColumn("revenue", F.col("qty") * F.col("unit_price"))
by_cat = (sales.groupBy("category")
.agg(F.count("*").alias("orders"),
F.sum("revenue").alias("total_revenue"),
F.round(F.avg("revenue"), 2).alias("avg_order"))
.orderBy(F.desc("total_revenue")))
by_cat.show()

OUTPUT (sample)

+----------+------+-------------+---------+
| category|orders|total_revenue|avg_order|
+----------+------+-------------+---------+
| Laptop| 1260| 494225000|392242.06|
| Mobile| 1263| 183377500| 145192.0|
| Watch| 1246| 67262500| 53982.74|
|Headphones| 1231| 25482000| 20700.24|
+----------+------+-------------+---------+

Verification: let us do the same calculation with Pandas (this OUTPUT is real). The results of both should be the same:

import pandas as pd
pdf = pd.read_csv("sales.csv")
pdf["revenue"] = pdf["qty"] * pdf["unit_price"]
res = (pdf.groupby("category")["revenue"]
.agg(orders="count", total_revenue="sum", avg_order="mean")
.round(2)
.sort_values("total_revenue", ascending=False))
print(res)
OUTPUT​
orders total_revenue avg_order
category
Laptop 1260 494225000 392242.06
Mobile 1263 183377500 145192.00
Watch 1246 67262500 53982.74
Headphones 1231 25482000 20700.24

By month and by city​

monthly = (sales
.groupBy(F.month("order_date").alias("month"))
.agg(F.sum("revenue").alias("revenue"))
.orderBy("month"))
monthly.show(4)
city_rev = (sales.groupBy("city")
.agg(F.sum("revenue").alias("revenue"))
.orderBy(F.desc("revenue")))
city_rev.show()

OUTPUT (sample)

+-----+--------+
|month| revenue|
+-----+--------+
| 1|69752000|
| 2|59793500|
| 3|60314000|
| 4|66487000|
+-----+--------+
+---------+---------+
| city| revenue|
+---------+---------+
| Lahore|171351000|
| Peshawar|159378000|
|Islamabad|151400500|
| Karachi|150230500|
| Quetta|137987000|
+---------+---------+

You can turn a small result into Pandas ( toPandas() ) and draw a chart. Careful: toPandas() brings all the data into the driver's memory, so run it only on small aggregated results , not on the whole big dataset.

import matplotlib.pyplot as plt
pdf_cat = by_cat.toPandas() # small result (4 rows)
pdf_mon = monthly.toPandas()
# then a bar chart and a line chart
with matplotlib (as in Module 2)
By month and by city: Revenue by category (million Rs.) Revenue by month (million Rs.) Laptop 494 68 Mobile 183 66 64 watch 67 z9 Headphones 25 500 100 200 300 400 10 11 12
By month and by city: Revenue by category (million Rs.) Revenue by month (million Rs.) Laptop 494 68 Mobile 183 66 64 watch 67 z9 Headphones 25 500 100 200 300 400 10 11 12

42.5 Spark SQL​

If you know SQL, you can turn a DataFrame into a temporary view and write SQL directly. The DataFrame and SQL run on the same engine , so the speed is the same.

sales.createOrReplaceTempView("sales")
top = spark.sql("""
SELECT customer_id,
COUNT(*) AS orders,
SUM(revenue) AS spent
FROM sales
GROUP BY customer_id
ORDER BY spent DESC
LIMIT 3
""")
top.show()

OUTPUT (sample)

+-----------+------+-------+
|customer_id|orders| spent|
+-----------+------+-------+
| 615| 11|3319500|
| 397| 9|3078000|
| 69| 14|3035500|
+-----------+------+-------+

42.6 Lazy Evaluation, Transformations and Actions​

The most important thing about Spark: when you write filter , select , withColumn , groupBy and other transformations , Spark does not run anything ; it only remembers a plan (DAG) . The work starts only when you run an action ( show , count , collect , write ).

Transformations (lazy)Actions (work starts)
select,filter/where,withColumnshow(),count()
groupBy().agg(),join,orderBycollect(),take(n),first()
repartition,distinct,dropwrite.parquet(...),toPandas()
42.6 Lazy Evaluation, Transformations and Actions: Lazy plan (DAG):it runs only whenan actionarrives read filter withColumn groupBy (narrow) (narrow) CSV (WIDE shuffle) Stage 1: partitions work separately, no data moved around...
42.6 Lazy Evaluation, Transformations and Actions: Lazy plan (DAG):it runs only whenan actionarrives read filter withColumn groupBy (narrow) (narrow) CSV (WIDE shuffle) Stage 1: partitions work separately, no data moved around...
Narrow vs Wide transformations:​
  • Narrow ( filter , select , withColumn ): each partition works on its own , and no data has to be sent between machines . Cheap and fast.

  • Wide ( groupBy , join , orderBy , distinct ): data has to be sent between partitions (shuffle) , which is expensive because of the network and disk. Spark breaks the plan into stages : a new stage at every wide transformation.

(sales.filter(F.col("qty") > 1)
.groupBy("city")
.agg(F.count("*").alias("n"))
.explain())

OUTPUT (sample; the detailed form of the plan varies with the version)

== Physical Plan == AdaptiveSparkPlan
isFinalPlan=false +- HashAggregate(keys=[city#2], functions=[count(1)]) +- Exchange hashpartitioning(city#2, 200) <-- this is the SHUFFLE +- HashAggregate(keys=[city#2], functions=[partial_count(1)]) +- Filter (isnotnull(qty#4) AND (qty#4 > 1)) +- FileScan csv [city#2,qty#4] ...

explain() If you see Exchange in the output, understand that a shuffle is happening there. Improving performance often means reducing shuffles.

42.7 Joins and Window Functions​

Join​

regions = spark.createDataFrame(
[("Karachi", "Sindh"), ("Lahore", "Punjab"), ("Islamabad", "Capital"),
("Quetta", "Balochistan"), ("Peshawar", "KP")],
["city", "region"])
joined = sales.join(regions, on="city", how="left")
(joined.groupBy("region")
.agg(F.sum("revenue").alias("revenue"))
.orderBy(F.desc("revenue"))
.show())

OUTPUT (sample)

+-----------+---------+
| region| revenue|
+-----------+---------+
| Punjab|171351000|
| KP|159378000|
| Capital|151400500|
| Sindh|150230500|
|Balochistan|137987000|
+-----------+---------+
Join typeResult
innerOnly the rows whose key is in both tables
leftAll rows of the left table; null if there is no match on the right
rightAll rows of the right table
outerAll rows of both
left_semi/left_antiThe rows of the left whose keyis/is notin the right (the right's columns do not come
through)

Window function: ranking within each group​

groupBy reduces the rows. A Window function keeps the rows but does calculations (rank, running total) within a group . For example: the city with the highest revenue in each category :

from pyspark.sql.window import Window
cc = (sales.groupBy("category", "city")
.agg(F.sum("revenue").alias("revenue")))
w = Window.partitionBy("category").orderBy(F.desc("revenue"))
(cc.withColumn("rank", F.row_number().over(w))
.filter(F.col("rank") == 1)
.orderBy("category")
.show())
OUTPUT (sample)​
+----------+------+---------+----+
| category| city| revenue|rank|
+----------+------+---------+----+
|Headphones|Quetta| 5437000| 1|
| Laptop|Lahore|110832500| 1|
| Mobile|Lahore| 40554000| 1|
| Watch|Lahore| 14556500| 1|
+----------+------+---------+----+

42.8 Performance: partitions, cache and Parquet​

TechniqueWhy and how
Filter early, select earlyReduce rows and columns before the heavy work (join/groupBy).
Built-in functions (F.*)Many times faster than a Python UDF, because Spark can optimize them.
Store in ParquetColumnar + compressed; reads only the columns it needs.
cache() / persist()Keep a DataFrame that is used again and again in memory.
repartition(n) / coalesce(n)repartitionsplits evenly with a shuffle;coalesce reducespartitions without a
shuffle.
Broadcast joinAvoid a shuffle when joining with a small table.
Avoid collect()Bringing all the data to the driver can use up its memory.
clean = sales.filter(F.col("qty") > 0).cache() # keep in memory
print(clean.count()) # first action: fills the cache
print(clean.count()) # second:
from the cache, fast
print(clean.rdd.getNumPartitions())
print(clean.repartition(8).rdd.getNumPartitions())
print(clean.coalesce(2).rdd.getNumPartitions())
clean.unpersist() # when the work is done, empty the cache

OUTPUT (sample; a small file often has few partitions) 5000 5000 1 8 1

Note: coalesce(2) can only reduce partitions; if there is already 1, it stays at 1.

Writing and reading Parquet​

(sales.write
.mode("overwrite")
.partitionBy("category") # a separate folder per category
.parquet("sales_parquet"))
fast = spark.read.parquet("sales_parquet")
# only the Laptop folder is read
fast.filter(F.col("category") == "Laptop").count()

OUTPUT (sample: the folder structure and the count) sales_parquet/ _SUCCESS category=Headphones/part-....snappy.parquet category=Laptop/part-....snappy.parquet category=Mobile/part-....snappy.parquet category=Watch/part-....snappy.parquet

1260

partitionBy("category") splits the data into folders , and when you filter on category, Spark reads only the necessary folder ( partition pruning ). Use partitionBy only on a column with few unique values (category, yearmonth); on a column like customer_id you would create thousands of tiny folders.

42.9 A short look at MLlib: groups of customers​

MLlib is Spark's machine learning library. Its Pipeline is like scikit-learn's, but the data stays distributed. Below is an example of splitting customers into 3 groups by their spending and orders ( KMeans , recall Module 4 Chapter 16).

from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.clustering import KMeans
from pyspark.ml import Pipeline
cust = (sales.groupBy("customer_id")
.agg(F.count("*").alias("orders"),
F.sum("revenue").alias("spent")))
assembler = VectorAssembler(inputCols=["orders", "spent"], outputCol="raw")
scaler = StandardScaler(inputCol="raw", outputCol="features")
kmeans = KMeans(k=3, seed=42, featuresCol="features")
model = Pipeline(stages=[assembler, scaler, kmeans]).fit(cust)
result = model.transform(cust)
result.groupBy("prediction").count().orderBy("prediction").show()

OUTPUT (sample; the group sizes may differ in your run) +----------+-----+ |prediction|count| +----------+-----+ | 0| ...| | 1| ...| | 2| ...| +----------+-----+

This is only a glimpse : the real benefit of MLlib comes when the data is so big that scikit-learn cannot run on one machine. For small data, scikit-learn is better and has more features.

42.10 Pandas vs Spark: which to use when?​

PandasSpark (PySpark)
Data sizeUp to one machine's RAM (typically GBs)GB to PB, on a cluster
Speed (small data)FastStart-up overhead, slow on small data
Lazy?No, runs immediatelyYes, runs on an action
DataIn memory, mutableDistributed, immutable (every step makes a new DataFrame)
Learning curveEasyA bit more (shuffle, partitions, plan)
WhenEDA, small projectsBig logs, pipelines, production ETL

42.11 Common Mistakes and Hands-on​

Hands-on​
  • RDD: in the word count, put in your own 5 sentences and keep only the words that occur 2 or more times ( filter ).

  • DataFrame: Count the rows where

qty >= 3

and

city == "Lahore"

.

  • New column: Create
revenue_lakh = revenue / 100000

and apply round(..., 2) .

  • groupBy: for each city and category pair, compute the average qty and show the top 5.

  • SQL: write the by_cat calculation above in SQL and match the result.

  • Window: find the best-selling category of each month ( row_number ).

  • Performance: In the output of explain() , look for Exchange . Does it appear in a filter-only query without groupBy ?

  • Parquet: Write with partitionBy("city") and count the folders.

KEY TAKEAWAYS​
  • Spark = an in-memory distributed engine: the Driver makes the plan, Executors run the tasks, the Cluster Manager allocates resources.

  • RDD is the foundation; DataFrame + Spark SQL are the everyday tools (fast thanks to the optimizer).

  • Lazy evaluation: transformations only build a plan (DAG); work happens on an action .

  • Narrow transformations are cheap; wide ones (groupBy, join, orderBy) shuffle and are expensive.

  • Write the schema yourself , filter/select early, use built-in functions, Parquet and cache .

  • For a join with a small table use a broadcast join ; use collect() and toPandas() only on small results.

  • For small data use Pandas , for big data use Spark .