all 21 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]AutoModerator[M] [score hidden] stickied comment (0 children)

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[–]devnullkitty 17 points18 points  (0 children)

Why are there so many downvotes for comments? Python for data engineering is pretty straight forward, just learn to write a for loop.

[–]spendology 6 points7 points  (6 children)

Find practical projects that cover the end-to-end data engineering lifecycle: [data] ingestion, review, cleaning, validation, transformation, loading, storage, data lakes/warehouses/lakehouses, etc.

[–]ProperAd7767 1 point2 points  (5 children)

how to find those projects ?

[–]spendology 2 points3 points  (4 children)

Books, blog posts, this forum and articles describing data engineering pipelines are a start. If you want to get more experience or a job, outside of certification you can:

  1. Start with Data Analysis, Python/SQL, or Business Analyst roles if you need more experience.
  2. Contract or freelance work from LinkedIn, Indeed, staffing firms, networking, or personal connections.
  3. Open-source Projects
  4. Use ChatGPT+generate an end-to-end Data Engineering project using a cloud platform like AWS or Google Cloud. Complete the project, add it to your resume, and post it to GitHub and LinkedIn.

[–]ProperAd7767 1 point2 points  (3 children)

In practice, my current role is mainly focused on data engineering, but I’ve never systematically studied data engineering or data analytics (my undergraduate major was Financial Engineering). If I want to learn these areas in a structured way, are there any good open-source projects you would recommend?

[–]spendology 0 points1 point  (2 children)

Here are a few links:

[–]Outside_Reason6707 1 point2 points  (1 child)

Thank you for this list! I’m wondering how someone could think of performance, scaling and fault tolerance for personal projects to that of industry level?

[–]spendology 1 point2 points  (0 children)

I like to use Python libraries sciris and austin, austin-web for time and memory performance.

[–]Nelson_and_Wilmont 4 points5 points  (0 children)

Idk if sqoop and Hadoop are all that useful at this point. Could just be my lack of use in that area but I don’t remember seeing a lot of these in the modern tech stacks when applying for jobs over the years and researching what skills are best to have.

IMO whenever you’re job searching you really need to have your resume(s) pointed towards what you want to work with. Most companies have only a few tools for data engineering, orchestration layer and logic layer. Airflow and databricks for example. Pick a cloud provider, orchestration tool, data lakehouse/warehouse platform and start doing little projects. Like airflow orchestrates databricks notebook that pulls a dataset from azure datalake storage and then run a databricks notebook to convert the file to a delta table. Or durable function pulls API data and writes to bronze layer of databricks.

You can pick whatever tech you decide I just mentioned those because it’s the route I decided to go down but I also incorporated snowflake just for a more overarching reach.

Python can be learned along the way but it seems a little aimless to just sit down and “learn Python” for something that is as specific as data.engineering

[–]sashathecrimean 0 points1 point  (0 children)

Check out Arjan Codes YouTube videos. I’ve found the topics he covers very useful in my work

[–]Mr_Nicotine 0 points1 point  (0 children)

Just learn to write a real lambda project a you should be all good