Skip to main content

Chapter 45: Home-Based Projects

MODULE 7 | WEEK 45​

Chapter 45: Home-Based Projects​

In the class projects, the instructor and classmates were there to help. In home-based projects you have to yourself choose the problem, find the data, build the pipeline and present the result alone . This is the work done in a real job , and it will become the most valuable part of your portfolio (GitHub). This chapter has 3 project options : you must complete at least one (the second/third are bonus ). For every option the problem, data, work milestones, deliverables and rubric are given.

TOPIC CHECKLIST (tick them off as you go)​
  • ☐ Rules for choosing a project and selecting data

  • ☐ Option A: Big Data Analytics (PySpark)

  • ☐ Option B: Cloud Data Pipeline (AWS or Azure)

  • ☐ Option C: Streaming Sensor Monitor (Structured Streaming)

  • ☐ A 5-day time plan

  • ☐ Deliverables, GitHub and the submission checklist

  • ☐ Module 8 summary and the next step

45.1 How to choose a project?​

  • Interest: choose a problem whose answer you yourself want to know (cricket, e-commerce, taxis, weather, cities). Interest is what gets a project finished.

  • Data size: big enough that the benefit of Spark shows (at least hundreds of thousands of rows or several files); but not so big that it will not run on Colab/the free tier.

  • Data source: open data portals, Kaggle or government/UN data. Read the license and terms.

  • Question first: decide "which question do I want to solve?", then pick the tools, not the other way round.

  • MVP: build the smallest working version first, then improve it.

45.2 Option A: Big Data Analytics (PySpark)​

45.3 Option B: Cloud Data Pipeline (AWS or Azure)​

Milestones:​
DayWork
1Account safety (MFA, budget), bucket/container and zones, data upload.
2Write the Spark job (cleaning + aggregates), run it locally and check it.
3Run the job on the cloud (or Colab + cloud storage); writeclean/andcurated/Parquet.
4Security test (read-only identity), lifecycle rule, cost calculation, (bonus: schedule/query).
5Architecture diagram, README, demo,cleanupand screenshot.
Rubric (Option B)Marks
------
Zones and pipeline work30
Proof of security (private, least privilege, no key in code)25
Cost control (budget, lifecycle, cleanup screenshot)15
Architecture diagram and explanation15
README and demo15
Total100

45.4 Option C: Streaming Sensor Monitor​

45.5 Deliverables and GitHub​

ItemRequired?Details
CodeYesA notebook or.pyfiles, clean and with comments
README.mdYesThe structure from Chapter 44.5: problem, data, approach, result, how
to run
Architecture diagramYesA diagram of the pipeline (draw.io/Excalidraw/PowerPoint is fine too)
Data quality / cost reportYesThe cleaning counts and (for cloud) the cost calculation
Demo recordingYesA 2-3 minute screen recording
requirements.txt / environmentBetterWith versions, so that someone else can run it too
.gitignoreYesData,.env, keys and large filesmust not be committed

45.6 Submission checklist​

  • ☐ The problem and questions are clear in 1-2 sentences

  • ☐ The data source and license are written down

  • ☐ Every cleaning step is documented

  • ☐ Spark results have been checked with another method (Pandas/SQL)

  • ☐ No password/key in the code or on GitHub

  • ☐ Cloud resources are deleted (screenshot)

  • ☐ Someone else can run the project from the README

  • ☐ The demo is under 3 minutes

45.7 Module 8 Summary​

ChapterMain ideaKey code / tool
38: Big Data5 Vs, scale out, HDFS blocks and replication, MapReduce,
Parquet, partitioning/skew, batch vs streaming
Python simulations, word count,
partition_sizes
39: Apache SparkDriver/Executors, RDD, DataFrame, SQL, lazy evaluation,
shuffle, join, window, Parquet
SparkSession,groupBy.agg,
Window,explain
40: Cloud (AWS/Azure)IaaS/PaaS/SaaS, regions, IAM, object storage, cost control,
data lake pipeline, ML deploy
boto3,azure-storage-blob,
S3/ADLS paths
41: Class ProjectsRetail ETL, Data Lake zones, Log analysis, README/demoPipeline, regexp_extract, Parquet
42: Home ProjectsChoosing your own problem, performance, cloud pipeline or
streaming, GitHub
End-to-end project

Next step: you now have a complete toolbox from Python through Machine Learning and Generative/Agentic AI to Big Data + Cloud . Going forward: (1) keep your projects on GitHub tidy and build a portfolio page , (2) take a cloud entry-level certificate (AWS Cloud Practitioner or Microsoft AZ-900/DP-900), (3) go deep into one of Data Engineering (Airflow, dbt, Delta Lake/Iceberg, Kafka) or ML Engineering (MLOps).

45.8 Common Mistakes and Hands-on​

Hands-on​
  • Write a self-check list of the four things: problem, data, cleaning, result; one sentence each.

  • Tidy up your GitHub profile : one pinned repository and one good README.

  • Explain your project in 2 minutes to a non-technical friend and see where they did not understand.

  • In the project, improve one thing that was not in the rubric (for example one more chart or a cost optimization).

KEY TAKEAWAYS​
  • Home project = yourself: problem, data, pipeline, result, README, demo .

  • Question first, tools later ; MVP first , then improvement.

  • Cleaning documented , results verified by another method .

  • Security and cost: no keys in code, budget alert, cleanup after the work.

  • Streaming: understanding window, watermark and output mode is essential.

  • GitHub + README + demo = your portfolio.