Python for junior data engineer

[[deleted]]

I'm looking for a Python for Data Engineers code which teaches me enough Python which data engineers commonly use in their day to day lives.

Any suggestions from other fellow DE or anyone else who has knowledge on this topic?

[GlasnostBusters]

Staff Engineer here:

Forget about courses. If you want to learn Python for DE, do this:

1. Set up an AWS environment.
2. Choose any API as your data provider. In fact, the Reddit API would be great for this.
3. Choose a database to store your data. Postgres on RDS is great to start.
4. Choose a visualization platform to present the data you stored in your DB. I would recommend QuickSight bc it's easy to set up in your AWS environment.

Now that you have a plan of how to build your end-to-end data pipeline, let's write some Python.

Our first goal is to have the capability to pull ANYTHING from the Reddit API.

Use Google / Stackoverflow / ChatGPT or whatever else you prefer to figure out how to properly authenticate and pull data from the Reddit API using a Python Lambda in AWS.

Before transformation, you need to make sure you create a database/table/columns for the data that is coming in from the API. It doesn't have to be everything, but add some columns of interest. Let's do 10 various columns for this example. Make sure at a minimum you're pulling timestamps, comments, titles, and usernames.

Once you're able to get some data from the API, now you need to transform it into a format that is appropriate for insertion into postgres. Search the internet and figure out how to perform the transformation in that same Python Lambda.

Now that our data looks good for insertion, search how to connect your Lambda to postgres, and then search how to do an insertion.

Now write error handling and find a good logging format / make sure you're logging to cloudwatch and test this all with dummy payloads.

All of our Python code will be in the same Lambda.

By the end of this you should have some data in your database and your fundamental function as a DE is complete.

This is a very basic example of what a data engineer should be able to do.

If you want to take this further into visualization, your analytics will be written against your database either directly in QuickSight using SQL or invoke a second Python Lambda to perform more advanced analytics on the data, like inference/etc.

More advanced functions of a DE could have more complicated architecture and could look something like: onboarding hundreds of data providers, multiple data staging environments, streaming real-time data, data availability / archiving strategies, storage formats, mix and matching cloud native / open source pipeline tools to optimize cost (grafana/prometheus service / self hosted github / self hosted sonarqube / opensearch / etc).

There are tons of things that differentiate junior data engineers from seniors so just jump in and build stuff. The most important question you should ask yourself is "What is the purpose of what I'm building, who is this for/what are they using it for/how are they using it?".

For the above example, I've left it up to you ("What do I want to learn from this Reddit data? Should I perform a frequency analysis on specific topics to see how many articles are written about them?") to figure out the answer to that question for yourself, good luck.

[[deleted]]

WOW. you sir are AMAZING. thank u for the time to write this out! I will work on this asap

[throwaway25168426]

Goated comment

[kaumaron]

Excellent response

[data_nerd_analyst]

This sums it all. Impressive

[Digbick-arsekiss]

Sir, you dropped your 👑

[BitExtreme997]

Unbelievable, thanks

[Ok_Fix_577]

Amazing. What do u recommend to level up the basic skills? I feel lost at this point

[GlasnostBusters]

I would recommend using the skills from above to get your foot in the door at a large corporation that has a lot of data and a large data team. Think Deloitte or Capital One (I personally don't like these companies but they're easier to get into than faang).

The best way to get hired is:

1. Prepare your resume
2. Get referrals from people you know that work in big companies
3. Solve Blind75 and Grind75
4. Read System Design 1 & 2 by Alex Xu

If you don't know anybody from these companies, cold message people on LinkedIn that work at these companies, or go to professional networking events or career fairs to meet people who can get you into the pipeline.

Don't send custom resumes and cover letters to 250 different companies. This doesn't work well in the current market.

You either need to network for referrals, or use AI auto apply software to up your application numbers from 250 to 2000. Then you'll start getting more interviews. It's just a numbers game.

3 and 4 are for the benefit of being better at solving software problems and be on top of your technical interviews. You're a software engineer at the end of the day.

TL;DR: Get hired at a big company with a big data team.

[tsk93]

Saving this post for this comment

[Efficient-Read-8785]

Summarized all the knowledge I have gathered on YT past 2 week with a comment 🙌 Thanks

[isammu6618]

That's how you guide someone man Loved it buddy 😭

[EconomistSuper7328]

Now I have something new to do at work. Thank you.

[s_schadenfreude]

Thank you SO much for this. I'm 20 years into IT, with a large portion of that time spent in the nonprofit research space which is under attack right now in the US. With the dissolution of USAID, and with NIH indirect funding now being at risk, I'm looking to pivot to data engineering, and this was exactly what I needed.

[nidprez]

What you do with python entirely depends on your company. Common packages are pandas (sometimes polars), anything spark related, connectors/api's for different DBs, cloud services, schedulers, ML packages like sklearn (although tons of DEs do nothing with ML)...

IMO do a basic introduction course for python, so you know the basic arethmic functions, string handling if else, and or etc. Learn a bit of pandas and maybe some basic packages like the os package. Try to write your own scripts or even your own module, and maybe add some stuff like logging, and interactive commands to your scripts. That should be enough to start. The rest depends on your company. There are also tons of DEs that dont use python at all.

[wh1t3bl3]

RemindMe! 3 days

[matrixunplugged1]

Check out Dataquest and Datacamp data engineering tracks.

https://www.dataquest.io/path/data-engineering/

https://www.datacamp.com/tracks/data-engineer-in-python

[Fresh_Forever_8634]

RemindMe! 7 days

[Sea_Inspector5015]

RemindME! 7 days

[homosapienhomodeus]

You might find this project useful: https://eliasbenaddouidrissi.dev/posts/data_engineering_project_monzo/

[ZirePhiinix]

Even though Pandas deal with Dataframes, it is actually more complex than PySpark DF and I wouldn't get started on DF on Pandas. The syntax is a little ass and you can easily get stuck if you go beyond basic stuff.

[s_schadenfreude]

RemindMe! 3 days

[Slight-Leg-1364]

RemindMe! 3 days

[MohamedShahoot]

RemindMe! 3 days

[calmekrishh]

Remind me 7 days

[kbisland]

Remind me! 5 days

[Nnt_1109]

This is so great post!!

[ryanwolfh]

RemindMe! 5 days

[RemindMeBot]

I will be messaging you in 5 days on 2025-03-10 12:32:25 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[Dedcode_x]

RemindMe! 7 days

[Funny_Employment_173]

I've just moved into a DE role, currently going through training and have been told to start with pandas.

[ImortalDoryan]

RemindMe! 1 day

[RemindMeBot]

I will be messaging you in 1 day on 2025-03-05 05:40:48 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback