Chapter 45: Home-Based Projects
MODULE 7 | WEEK 45
Chapter 45: Home-Based Projects
In the class projects, the instructor and classmates were there to help. In home-based projects you have to yourself choose the problem, find the data, build the pipeline and present the result alone . This is the work done in a real job , and it will become the most valuable part of your portfolio (GitHub). This chapter has 3 project options : you must complete at least one (the second/third are bonus ). For every option the problem, data, work milestones, deliverables and rubric are given.
TOPIC CHECKLIST (tick them off as you go)
-
☐ Rules for choosing a project and selecting data
-
☐ Option A: Big Data Analytics (PySpark)
-
☐ Option B: Cloud Data Pipeline (AWS or Azure)
-
☐ Option C: Streaming Sensor Monitor (Structured Streaming)
-
☐ A 5-day time plan
-
☐ Deliverables, GitHub and the submission checklist
-
☐ Module 8 summary and the next step
45.1 How to choose a project?
-
Interest: choose a problem whose answer you yourself want to know (cricket, e-commerce, taxis, weather, cities). Interest is what gets a project finished.
-
Data size: big enough that the benefit of Spark shows (at least hundreds of thousands of rows or several files); but not so big that it will not run on Colab/the free tier.
-
Data source: open data portals, Kaggle or government/UN data. Read the license and terms.
-
Question first: decide "which question do I want to solve?", then pick the tools, not the other way round.
-
MVP: build the smallest working version first, then improve it.
45.2 Option A: Big Data Analytics (PySpark)
45.3 Option B: Cloud Data Pipeline (AWS or Azure)
Milestones:
| Day | Work |
|---|---|
| 1 | Account safety (MFA, budget), bucket/container and zones, data upload. |
| 2 | Write the Spark job (cleaning + aggregates), run it locally and check it. |
| 3 | Run the job on the cloud (or Colab + cloud storage); writeclean/andcurated/Parquet. |
| 4 | Security test (read-only identity), lifecycle rule, cost calculation, (bonus: schedule/query). |
| 5 | Architecture diagram, README, demo,cleanupand screenshot. |
| Rubric (Option B) | Marks |
| --- | --- |
| Zones and pipeline work | 30 |
| Proof of security (private, least privilege, no key in code) | 25 |
| Cost control (budget, lifecycle, cleanup screenshot) | 15 |
| Architecture diagram and explanation | 15 |
| README and demo | 15 |
| Total | 100 |
45.4 Option C: Streaming Sensor Monitor
45.5 Deliverables and GitHub
| Item | Required? | Details |
|---|---|---|
| Code | Yes | A notebook or.pyfiles, clean and with comments |
| README.md | Yes | The structure from Chapter 44.5: problem, data, approach, result, how to run |
| Architecture diagram | Yes | A diagram of the pipeline (draw.io/Excalidraw/PowerPoint is fine too) |
| Data quality / cost report | Yes | The cleaning counts and (for cloud) the cost calculation |
| Demo recording | Yes | A 2-3 minute screen recording |
| requirements.txt / environment | Better | With versions, so that someone else can run it too |
| .gitignore | Yes | Data,.env, keys and large filesmust not be committed |
45.6 Submission checklist
-
☐ The problem and questions are clear in 1-2 sentences
-
☐ The data source and license are written down
-
☐ Every cleaning step is documented
-
☐ Spark results have been checked with another method (Pandas/SQL)
-
☐ No password/key in the code or on GitHub
-
☐ Cloud resources are deleted (screenshot)
-
☐ Someone else can run the project from the README
-
☐ The demo is under 3 minutes
45.7 Module 8 Summary
| Chapter | Main idea | Key code / tool |
|---|---|---|
| 38: Big Data | 5 Vs, scale out, HDFS blocks and replication, MapReduce, Parquet, partitioning/skew, batch vs streaming | Python simulations, word count, partition_sizes |
| 39: Apache Spark | Driver/Executors, RDD, DataFrame, SQL, lazy evaluation, shuffle, join, window, Parquet | SparkSession,groupBy.agg,Window,explain |
| 40: Cloud (AWS/Azure) | IaaS/PaaS/SaaS, regions, IAM, object storage, cost control, data lake pipeline, ML deploy | boto3,azure-storage-blob,S3/ADLS paths |
| 41: Class Projects | Retail ETL, Data Lake zones, Log analysis, README/demo | Pipeline, regexp_extract, Parquet |
| 42: Home Projects | Choosing your own problem, performance, cloud pipeline or streaming, GitHub | End-to-end project |
Next step: you now have a complete toolbox from Python through Machine Learning and Generative/Agentic AI to Big Data + Cloud . Going forward: (1) keep your projects on GitHub tidy and build a portfolio page , (2) take a cloud entry-level certificate (AWS Cloud Practitioner or Microsoft AZ-900/DP-900), (3) go deep into one of Data Engineering (Airflow, dbt, Delta Lake/Iceberg, Kafka) or ML Engineering (MLOps).
45.8 Common Mistakes and Hands-on
Hands-on
-
Write a self-check list of the four things: problem, data, cleaning, result; one sentence each.
-
Tidy up your GitHub profile : one pinned repository and one good README.
-
Explain your project in 2 minutes to a non-technical friend and see where they did not understand.
-
In the project, improve one thing that was not in the rubric (for example one more chart or a cost optimization).
KEY TAKEAWAYS
-
Home project = yourself: problem, data, pipeline, result, README, demo .
-
Question first, tools later ; MVP first , then improvement.
-
Cleaning documented , results verified by another method .
-
Security and cost: no keys in code, budget alert, cleanup after the work.
-
Streaming: understanding window, watermark and output mode is essential.
-
GitHub + README + demo = your portfolio.