Module 8 Test: 20 MCQs
This is your progress tracker : 20 questions, 5 Easy, 10 Moderate, 5 Hard . First solve the questions yourself (for code questions, think of the output before running it in Colab), then check against the Answer Key.
| Score | Meaning |
|---|---|
| 18 to 20 | Excellent! Module 8 is your strong foundation. |
| 12 to 17 | Good. Re-read the chapters where you made mistakes. |
| Below 12 | Don't worry. Re-read Chapters 41 to 45, practise with the projects, then take the test again. |
EASY
Q1. Which of the 5 Vs of Big Data tells how fast the data is arriving?
A) Volume B) Variety
C) Veracity
D) Velocity
Q2. In HDFS, how many copies (replicas) of each block are kept by default?
A) 3
B) 1
C) 2
D) 5
Q3. Which is the correct order of the three steps of MapReduce?
A) Reduce, Map, Shuffle
B) Shuffle, Map, Reduce
C) Map, Shuffle, Reduce
D) Map, Reduce, Shuffle
Q4. Which of the following is a Spark action?
A) filter()
B) count()
C) select()
D) withColumn()
Q5. In the cloud, what does pay-as-you-go mean?
A) Paying the whole year's bill in advance
B) The cloud is always free
C) Only big companies can use it
D) You pay only for what you use
MODERATE
Q6. 2 TB of data, each disk reads at 200 MB/second, and the data is spread evenly over 20 disks (1 TB = 1024 x 1024 MB). Roughly how long will it take to read all the data?
A) About 52 seconds
B) About 524 seconds
C) About 5243 seconds
D) About 10486 seconds
Q7. A 1000 MB file, block size 128 MB, replication 3. How many blocks will be made and how many block copies will be stored in total?
A) 8 blocks, 24 copies B) 7 blocks, 21 copies C) 8 blocks, 8 copies D) 8 blocks, 3 copies
Q8. What will be the output of this code?
from collections import defaultdict
g = defaultdict(list)
for line in ["a b a", "b a"]:
for w in line.split():
g[w].append(1)
print({k: sum(v) for k, v in g.items()})
A) {"a": 2, "b": 3} B) {"a": 5, "b": 5} C) {"a": 1, "b": 1} D) {"a": 3, "b": 2}
Q9. If the Spark explain() plan shows Exchange, what does this mean?
A) Data is being kept in the cache B) The file is being written as Parquet C) Data is being sent between partitions (shuffle)
D) The schema is being changed
Q10. In this Spark code, up to **
print
, the actual calculation (processing of data) was done on how many rows?**
df2 = df.filter(F.col("qty") > 2)
df3 = df2.withColumn("x", F.col("qty") * 2)
print("ready")
A) On all rows, because the filter ran
B) On none, because no action ran
C) Only on the rows where qty > 2 is true
D) Only on the first 20 rows
Q11. The partition rule is key % 4 . If the keys are 10 to 19 (10 keys), how many keys go to partition 2?
A) 3
B) 2
C) 4
D) 1
Q12. A table has 1 million rows and 20 columns. The query needs only 2 columns. Compared with a row-store, roughly how many fewer values does a column-store (Parquet) have to read?
A) 2 times fewer B) 20 times fewer C) 10 times fewer (2M compared with 20M) D) No difference
Q13. A cluster has an imaginary price of $0.50 per node per hour. 3 nodes run 10 hours a day for 20 days. What is the total compute cost?
A) $30 B) $150 C) $3,000 D) $300
Q14. A user's IAM policy allows only s3:GetObject (on the objects of one bucket). What can that user do?
A) Can both read and upload files
B) Can read files in that bucket, but cannot upload new files or delete
C) Can only delete
D) Can do nothing
Q15. The sales table has 5000 rows. In the regions table, Quetta is missing out of the 5 cities (only 4 cities). After sales.join(regions, "city", "left") , how many rows will there be, and what will be the region of the Quetta rows? (Quetta has 975 rows.)
A) 4025 rows, the Quetta rows disappear
B) 5001 rows
C) 5000 rows, Quetta's region is null
D) An error
HARD
Q16. A join is very slow because one customer_id has 90% of the records and the other partitions sit idle. Which is the most suitable solution?
A) Salting, or handle the skewed key separately (or broadcast if one table is small)
B) Increase the RAM of all executors in the cluster
C) Call cache() on the result
D) Use coalesce(1) to put everything in one partition
Q17. The times (in seconds) of the streaming events are below, and there is a 5-second tumbling window. Which window has the most events and how many?
events = [1, 3, 4, 7, 8, 11, 12, 13, 14, 14] # (second)
window = 5
A) [0s, 5s): 3 events
B) [5s, 10s): 5 events
C) [10s, 15s): 4 events
D) [10s, 15s): 5 events
Q18. In a data lake, why is it wrong to overwrite/modify the Raw zone?
A) Because the files in the Raw zone must be Parquet
B) Because the original data is lost and you cannot reprocess if there is a cleaning mistake
C) Because the Raw zone holds only aggregates
D) Because overwriting reduces the bill
Q19. A Spark job runs only 2 hours every night, but the cluster stays on 24x7 and the bill is very high. What is the best change?
A) Add more nodes to the cluster
B) Convert the data to CSV
C) Transient/auto-terminate cluster or serverless: runs only while the work is done
D) Make the bucket public
Q20. What will be the result (k, v) of this Spark code? ( row_number counts within each k in descending order of v.)
data = [("A", 10), ("A", 30), ("B", 5), ("B", 50), ("B", 20)]
df = spark.createDataFrame(data, ["k", "v"])
w = Window.partitionBy("k").orderBy(F.desc("v"))
(df.withColumn("r", F.row_number().over(w))
.filter("r = 2").select("k", "v").show())
A) ("A", 10) and ("B", 20) B) ("A", 30) and ("B", 50) C) ("A", 10) and ("B", 5) D) ("A", 30) and ("B", 20)
Module 8 Answer Key (Answers and explanations)
| # | Level | Answer | Explanation |
|---|---|---|---|
| 1 | Easy | D) Velocity | Velocity is the speed of data (events arriving every second). Volume is the amount, Variety is the kinds, Veracity is about trust. |
| 2 | Easy | A) 3 | The default replication factor is 3. That is why data survives even when one or two machines fail. |
| 3 | Easy | C) Map, Shuffle, Reduce | First Map (make key, value), then Shuffle (same keys in one place), and finally Reduce (sum/count). |
| 4 | Easy | B)count() | count()is an action: it runs the work immediately. filter, select,withColumn are transformations, which only build a plan. |
| 5 | Easy | D) You pay only for what you use | In pay-as-you-go you pay only for what you actually used (time, storage, data). |
| 6 | Moderate | B) About 524 seconds | Total MB = 2 x 1024 x 1024 = 2,097,152. Speed = 200 x 20 = 4000 MB/s. Time = 2,097,152 / 4000 = 524.3 seconds (about 8.7 minutes). |
| 7 | Moderate | A) 8 blocks, 24 copies | 1000 / 128 = 7.8, i.e. 8 blocks (the last one is smaller). Each block has 3 copies: 8 x 3 = 24. |
| 8 | Moderate | D) {"a": 3, "b": 2} | "a b a" and "b a" contain a three times (2 + 1) and b twice (1 + 1). Shuffle gathers the same key, Reduce sums them. |
| 9 | Moderate | C) Data is being sent between partitions (shuffle) | Exchange indicates a shuffle (wide transformations such as groupBy, join, orderBy). It is expensive, so the aim in performance work is to reduce shuffles. |
| 10 | Moderate | B) On none, because no action ran | filter and withColumn are transformations (lazy). Until an action like count/show/write runs, Spark only builds a plan. |
| 11 | Moderate | A) 3 | The keys with remainder 2: 10, 14, 18, i.e. 3. The rest: partition 0: 12, 16 (2); partition 1: 13, 17 (2); partition 3: 11, 15, 19 (3). |
| 12 | Moderate | C) 10 times fewer (2M compared with 20M) | In a row-store every row has to be read in full: 20 x 1M = 20M values. In a column-store only 2 columns: 2M. That is, 10 times fewer. |
| 13 | Moderate | D) $300 | 0.50 x 3 x 10 x 20 = 300 dollars. |
| 14 | Moderate | B) Can read files in that bucket, but cannot upload new files or delete | GetObject is permission only to read objects. PutObject (upload) or DeleteObject are separate permissions that were not given here (least privilege). |
| 15 | Moderate | C) 5000 rows, Quetta's region is null | A left join keeps all rows of the left table, so 5000. Where the key is not found on the right, the right-hand columns are null. An inner join would leave 4025 rows. |
| 16 | Hard | A) Salting, or handle the skewed key separately (or broadcast if one table is small) | The problem is data skew: the load is on one partition. More RAM or cache does not spread the load; coalesce(1) makes it worse. Salting splits the key up. |
| 17 | Hard | D) [10s, 15s): 5 events | Windows: [0,5): 1, 3, 4 = 3 events; [5,10): 7, 8 = 2; [10,15): 11, 12, 13, 14, 14 = 5. The most is 5, in [10s, 15s). |
| 18 | Hard | B) Because the original data is lost and you cannot reprocess if there is a cleaning mistake | Raw should stay exactly as it arrived. If a mistake is found in the cleaning or calculation, Clean and Curated can be rebuilt from Raw. Parquet/aggregates belong to Clean/Curated. |
| 19 | Hard | C) Transient/auto-terminate cluster or serverless: runs only while the work is done | The cost depends on how long the cluster runs. Creating/stopping the cluster only when the work happens (or using serverless) lowers the bill. Adding nodes raises the bill; a public bucket is dangerous. |
| 20 | Hard | A) ("A", 10) and ("B", 20) | In each k, v is in descending order: A: 30, 10; B: 50, 20, 5. |
r = 2
is
the second row: 10 for A and 20 for B.|