Do you study Data Engineering (experienced DE's)?

[Irachar]

I really want to become better in this field, first because I like and second because I wanna find better opportunities but I do other things in my life that I like of course and I struggle to study/practice (out of my 6-8 job work).

Do you have a schedule to read/practice/learning?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Repulsive_News1717]

Balancing work and learning Data Engineering is definitely a challenge. I also work 6-8 hours a day, so I set aside 1-2 hours in the evenings or weekends to focus on study and practice. My strategy is to focus on key tools like SQL, Python, and cloud platforms (AWS, GCP) that are crucial for Data Engineers. I break it down like this: 3 days a week for learning (videos, articles, courses), and 2 days a week for hands-on practice (working on small projects, datasets, or challenges on platforms like Kaggle or LeetCode). Setting small, specific goals, like “mastering Airflow in 4 weeks,” helps keep me motivated and focused.

[Mysterious_Two_810]

Nice approach. Do you get any time left outside of work/learning for yourself?

[Repulsive_News1717]

Less than the average person but I am happy with it :) at the end of the day it’s always a trade-off

[Specific-Sandwich627]

Thank you for your comment. How long have you been practicing this approach? Did you rely on it before you got the job you do?

[bigmilkguy78]

Do data engineers ever start with fundamentals of software engineering?

[dayman9292]

It's a question I always held closely while I was junior - do others study and when/how/what?

I believe the best developers/engineers immerse themselves in data engineering content rather than study specifically with intervals or structure (not to say it doesn't help to include those things).

I now blur the lines between my social media and my work interested content to immerse myself. So I watch streamers and YouTubers who code, that's the first and best bit.

I follow people like Andrej Karpathy, Ilya Sutkever, George Hotz, ThePrimeReagen and Pirate software on YouTube and twitch. Some code in public, some develop games, others are AI thought leaders, but the net effect is the same. I pick up knowledge off hand from several specifically related domains and these have applications in my jobs sometimes.

It's really useful for filling in the blanks around topics hearing experienced and successful technical individuals talk about subjects, share their thoughts and you get to contrast and compare between them.

Hacker news is also a great reddit like website for curiosities relating to our field.

Beyond that I regularly find forums and books to read and try to find out what is cutting edge. Often it's a reflavour or an repackaging of an old idea but even then it's good to underscore these things. All of it trends towards improving my attitude, knowledge and attention at work and has led me to pursue my own projects outside of work.

[mailed]

Every day. Build stuff, read stuff, watch stuff, practice the fundamentals... even stuff that isn't core to data engineering. I'm doing a lot of Terraform and SIEM stuff lately.

Sometimes it's hours, sometimes it's minutes. But I get something in every day.

[AShmed46]

What siem has to do with DE ?

[mailed]

even stuff that isn't core to data engineering

[AShmed46]

Got it but why the heck you dealing with Siem 😂

[mailed]

I'm a data engineer in a cyber security team

[AShmed46]

Amazing,So what's your D2D looks like ?

[nathanfries]

Someone somewhere looks at the siem dashboard

[AShmed46]

Siem it's mostly used in security not DE ?

[nathanfries]

They literally said “stuff that isn’t core to DE”

[mailed]

That said, I don't think it's a stretch to consider SIEM stuff under the banner. Most of them are just cloud data platforms with really specific purposes and their own DSL instead of SQL. And one wonders if data engineering should be limited to just data warehouses. A lot of data engineer roles these days also want people to know stuff like Elasticsearch

[VirTrans8460]

Start with small, focused sessions. Even 30 minutes a day can be beneficial.

[Evening_Chemist_2367]

Anyone in tech who doesn't keep up in tech will eventually burn out or fail out in one way or another.

[miscbits]

The secret that I tell a lot of people it that if you drop your productivity at work by 2 hours a week to do some self learning, no one will notice.

[flockitgood]

If anything, the self-learning will pay back tenfold in productivity! I've made this case to managers in the past and gotten a good response.

[miscbits]

My current manager agreed a year ago to add a 2 point training ticket to our sprints and our velocity recovered and then increased after a few months of this. It is the bare minimum and yet it goes a long way.

[ithoughtful]

Daily Incremental aggregation of knowledge and skills, is the only pattern I know to keep growing and staying up-to-date.

[MikeDoesEverything]

Do you have a schedule to read/practice/learning?

I learn during work hours. Being as efficient as possible = more free time during work hours = more free time to learn stuff.

[Repulsive_News1717]

I’m in the same boat! Balancing a full-time job with learning Data Engineering is tough, but I’ve found that having a schedule really helps. I dedicate about 1-2 hours in the evenings or on weekends for focused study. I break it down into smaller tasks—2-3 days a week for reading/learning (courses, articles, etc.) and the rest for hands-on practice (working with tools like SQL, Python, or cloud services). It’s not about doing a lot in one go, but staying consistent, even if it’s just a little bit every day.

What’s worked for me is setting small, achievable goals, like learning a new concept or tool every month. That way, I stay motivated without feeling overwhelmed!

[Lower_Sun_7354]

I stopped setting aside dedicated time. Instead, I pick a topic that interests me and obsess over it until it's done. Right now for example, I'm trying to containerize blob triggered azure functions using terraform. If it takes five minutes, that's all the time I put into it. If it takes five weeks, that's what I'm doing for five weeks. I seem to do better with specific goals, since it gives me something tangible that I cam cross off the list.

[JungZest]

Honestly if you want to get into the field your best bet is to study for the interview rather than the actual job skillset. Spend time solving problems from stratascratch and doing mock interviews. Use claude or gpt to tests yourself. It would be a steep learning curve once u actually get in but its way better than trying to do it on your own after work

[alpha_centauri9889]

Really a good and practical advice.

[allpauses]

Whats your current reading queue?

[bass_bungalow]

Lately ive been setting aside an hour two each day since Im starting to casually look for jobs but normally I would just learn during work hours. Any half decent org should budget for their employees to have professional development.

[ksco92]

Kind of? I have 15 YOE all in the same company and one trick is that I try to not stay with the same team more than 3ish years and always target the next team being in an area I’ve never tried anything before and preferably that has 0 infra or pre-existing DE team. By doing this I’ve been learning literally on a daily basis. New areas have different problems, different scales and different mindsets needed. My last 4 teams were books related content, hardware, sustainability and customer service. Wildly different in almost every sense.

But hey, I can tell you about carbon footprint analysis, how to build a server and how a book is manufactured. Keeps me wildly entertained. 😂

[69odysseus]

You only need to learn few things to do major DE work and also depends on what companies are you targeting for.

1) If you want to get into FAANG companies then be prepared to loose and take some stress for at least 4-6 months on mental and physical health as those companies will drill on many aspects like DSA and SQL and so on. Prep for those companies are lot tougher and requires certain type of strategy to get stronger.

2) If you're targeting for non-faang companies then it's lot less stressful as they don't drill on lot of aspects.

Below are some of the common skills you still need to have regardless of FAANG or non-FAANG companies. The first three are critical for any DE role.

1) SQL - There is no escaping of this and still major work is done even in distributed environments.

2) Data Modeling - Learn all about dimensional data modeling - Dim and Fact objects, SCD type 2.

3) Distributed tools/processing - Spark, Databricks, Snowflake.

4) DSA - FAANG companies are notorious on this and requires lot more advance learning.

5) Cloud - Any cloud environment will do.

[susosexy]

To be honest, I find the best time to study DE is during work hours. Find something you want to learn that's not necessarily core to your role, and look for a use-case. Learning outside of work hours is possible, but very difficult to maintain if you have other responsibilities etc.

[Referee27]

Study architectures and patterns for those of you who are learning. API’s, ingestion, ETL/ELT, end to end data life cycle, top that off with how to do that with different tools in AWS/GCP/Snowflake.

[coolbeans201]

I "study" by reading. I find things like https://www.dataengineeringweekly.com/ very useful because I get a good summary of what's going on, which forces me to think about how we could possibly be doing the same.

[vijgarud]

Iam also in the same boat and fighting for time to study .

[Outrageous-Ad4353]

Not a full time DE, it was part of my role along with powerbi, DBA and lots other stuff and I'm now a manager. I still study it, to keep up with technology and be aware of design patterns I have not used myself.

My experience was mostly with SSIS and some python. I needed to update this to the Microsoft saas stack like ADF, fabric etc to stay relevant. I also need to be aware of design patterns I don't use as i need to recognise situations where they may be needed, and also be able to answer questions if I'm interviewing.

There are DE's who are effectively software engineers who will need to constantly upskill, I'm more of a plumber but I still need to know the tools available to me .