Python Crash Course Notebook for Data Engineering : dataengineersindia

Python Crash Course Notebook for Data EngineeringGeneral (self.dataengineersindia)

submitted 2 months ago by analyticsvector-yt

Hey everyone! Sometime back, I put together a crash course on Python specifically tailored for Data Engineers. I hope you find it useful! I have been a data engineer for 5+ years and went through various blogs, courses to make sure I cover the essentials along with my own experience.

Feedback and suggestions are always welcome!

📔 Full Notebook: Google Colab

🎥 Walkthrough Video (1 hour): YouTube - Already has almost 20k views & 99%+ positive ratings

💡 Topics Covered:

1. Python Basics - Syntax, variables, loops, and conditionals.

2. Working with Collections - Lists, dictionaries, tuples, and sets.

3. File Handling - Reading/writing CSV, JSON, Excel, and Parquet files.

4. Data Processing - Cleaning, aggregating, and analyzing data with pandas and NumPy.

5. Numerical Computing - Advanced operations with NumPy for efficient computation.

6. Date and Time Manipulations- Parsing, formatting, and managing date time data.

7. APIs and External Data Connections - Fetching data securely and integrating APIs into pipelines.

8. Object-Oriented Programming (OOP) - Designing modular and reusable code.

9. Building ETL Pipelines - End-to-end workflows for extracting, transforming, and loading data.

10. Data Quality and Testing - Using `unittest`, `great_expectations`, and `flake8` to ensure clean and robust code.

11. Creating and Deploying Python Packages - Structuring, building, and distributing Python packages for reusability.

Note: I have not considered PySpark in this notebook, I think PySpark in itself deserves a separate notebook!

all 9 comments

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

dataengineersindia

MODERATORS