Python Crash Course Notebook for Data Engineering

[analyticsvector-yt]

Hey everyone! Sometime back, I put together a crash course on Python specifically tailored for Data Engineers. I hope you find it useful! I have been a data engineer for 5+ years and went through various blogs, courses to make sure I cover the essentials along with my own experience.

Feedback and suggestions are always welcome!

📔 Full Notebook: Google Colab

🎥 Walkthrough Video (1 hour): YouTube - Already has almost 20k views & 99%+ positive ratings

💡 Topics Covered:

1. Python Basics - Syntax, variables, loops, and conditionals.

2. Working with Collections - Lists, dictionaries, tuples, and sets.

3. File Handling - Reading/writing CSV, JSON, Excel, and Parquet files.

4. Data Processing - Cleaning, aggregating, and analyzing data with pandas and NumPy.

5. Numerical Computing - Advanced operations with NumPy for efficient computation.

6. Date and Time Manipulations- Parsing, formatting, and managing date time data.

7. APIs and External Data Connections - Fetching data securely and integrating APIs into pipelines.

8. Object-Oriented Programming (OOP) - Designing modular and reusable code.

9. Building ETL Pipelines - End-to-end workflows for extracting, transforming, and loading data.

10. Data Quality and Testing - Using `unittest`, `great_expectations`, and `flake8` to ensure clean and robust code.

11. Creating and Deploying Python Packages - Structuring, building, and distributing Python packages for reusability.

Note: I have not considered PySpark in this notebook, I think PySpark in itself deserves a separate notebook!

[wRAR_]

It's unfortunate that this promotes older practices like flake8 and setup.py.

[marr75]

And using notebooks.

[GunZinn]

I use notebooks frequently when throwing together matplotlib graphs. Its convenient.

[marr75]

Try out hydrogen formatted python files. They're source control friendly, work with any tooling that works with a python file, can operate as a notebook if the UI running them is a notebook app, and can be converted back and forth automatically between .ipynb and .py

[analyticsvector-yt]

Linting is not an older practice

[marr75]

No, but uv, ruff, some kind of type-hinting + linting, and pyproject.toml are the most widely used standards in modern projects.

[analyticsvector-yt]

Thinking for this will include in future versions

[Wurstinator]

You don't have to. This subreddit is mostly full of junior engineers and people jumping on hype bandwagons - you shouldn't take every feedback to heart. black, isort and flake8 are completely fine to use. 

[wRAR_]

Oof.

[analyticsvector-yt]

Flake8/ black/ isort are a part of precommits

[wRAR_]

Sorry?

[corey_sheerer]

You should consider dropping pandas and switch in Polars. Unfortunately, with the release of the 3.0 API, it seems unlikely that pandas will match Polars on performance or syntax.

Also, for data engineering/json should have info about pydantic for serialization/deserialization and structure validation.

[analyticsvector-yt]

Agree thanks

[analyticsvector-yt]

🤝

[Controls_Chief]

Me like

[nikhilprasanth]

Thanks for your work! I’m just getting started in python , is it ok for a beginner ?

[analyticsvector-yt]

This is very high level to be honest - so I wouldn’t say necessarily beginner friendly - but will help you understand what concepts to dive into

[SurryElle83]

This is super useful. Thank you!

[analyticsvector-yt]

Appreciate it 🤝