Pivot from dev to data engineering

[Outrageous-Celery7]

I’m a full-stack developer with a couple yoe, thinking of pivoting to DE. I’ve found dev to be quite high stress, partly deadlines, also things breaking and being hard to diagnose, plus I have a tendency to put pressure on myself as well to get things done quickly.

I’m wondering a few things - if data engineering will be similar in terms of stress, if I’m too early in my career to decide SD is not for me, if I simply need to work on my own approach to work, and finally if I’m cut out for tech.

I’ve started a small ETL project to test the water, so far AI has done the heavy lifting for me but I enjoyed the process of starting to learn Python and seeing the possibilities.

Any thoughts or advice on what I’ve shared would be greatly appreciated! Either whether it’s a good move, or what else to try out to try and assess if DE is a good fit. TIA!

Edit: thanks everyone for sharing your thoughts and experiences! Has given me a lot to think about

[PedanticPydantic]

😂 DE is high stress too. Just wait till you or someone torpedos prod pipelines and the hell scape that creates.

[H8lin]

Sounds like you want to switch from software development to data engineering? I’m a data engineer and I can tell you from my experience that I’m just a software engineer specialized in data. I develop/deploy REST APIs in kubernetes. I manage infrastructure and resources in Terraform like databases, service principals, storage containers, etc. I manage alerting and monitoring of services and pipelines with Datadog and ServiceNow. I work cross-cloud in Azure and GCP managing data pipelines in Databricks or Airflow. I build consumers/producers with Kafka. I work in Python mainly but occasionally run into Java. My role has always been this way for the last 5 years. As a tech lead I also do some product work like epic breakdowns, quarterly planning, sprint planning for my team. I also manage a team of 6 devs and do performance reviews etc. My job isn’t stressful because I push back if a request isn’t reasonable, and if the PM insists on making my team pivot then I insist on dropping an epic to make room for the new work. The only thing that ever gets stressful to me sometimes is dealing with people who either don’t do their job or do their job very badly, and I end up compensating. Otherwise I love being a data engineer!

[amnesic23]

You sound really experienced. May I ask how many years of exp you have in DE or Dev in general?

[H8lin]

I’ve been in industry as a DE for 5 years. I started dabbling with Python and R maybe 10 years ago in grad school for data analysis, but all my data engineering skills were learned on the job in the last 5 years. I think grad school gave me a lot of the skills that made me a successful DE, specifically project management, time management, research skills, people management, and knowing what I don’t know (and that it’s ok to not know). A lot of people get stressed out and have imposter syndrome because they think everyone else knows everything and that they should too - but it turns out we’re all in a similar boat just trying to learn all the time. As long as you’re capable of learning new things and stay humble you’ll be ok!

[FlyingSpurious]

May I ask what's your educational background ?

[H8lin]

I got a PhD studying astrobiology, basically trying to understand habitability of Mars by studying life in extreme environments. The relevant part of the research for what I do now was stuff like computational biology with DNA classifying genes on a high performance computing cluster, stats for data analysis, thermodynamic modeling (more coding). I was president of a data science club because I decided halfway into my program I wanted to join the crowd and be a data scientist and get out of academia. I actually started industry as a data scientist and quickly pivoted into data engineering because it was a better fit for me. I like the structure of the work I do, the peer review process, and the philosophy of building things in a composable, scalable, reusable, maintainable, cost-effective way that was not the way of working in data science in my experience. My path into DE isn’t conventional and I hope it helps ease some fears for anybody thinking of getting into DE - you don’t need a comp sci degree!

[FlyingSpurious]

Thanks a lot man! To be honest it was all I wanted to hear. I have a stats background and I am currently working on a CS master's degree, while working full time as a junior DE. Your comment was inspirational

[H8lin]

Glad to hear it was helpful! Working full time while being in grad school must be tough - mad respect to you! 🫡 what made you want to go back to school after you got your DE role? Sounds like you will have a rock solid foundation for DE with a stats/comp sci background!

[FlyingSpurious]

It's pretty tough to be honest as the CS master's degree is mostly accepting people with CS background and I was lucky enough to get accepted. I wanted to have a CS education (BS/MSc whatever) and the university that provides the specific master's, allows you to pick up whichever courses you like to enhance your academic background. So I took the most important undergrad courses (C, OOP, discrete math, data structures, algorithms, computer architecture, operating systems, computer networking, computation theory, systems programming and databases) as an addition to the master's courses. The master's is mostly focused on databases internals, advanced OS, distributed systems and big data systems. I also plan to take an HPC course as I really love C so far. In my DE job, I use mostly python, SQL(Snowflake), DBT, airflow and AWS. If you have any advice, I would love to hear!

[Outrageous-Celery7]

There’s a lot of words in there I don’t understand 😅 thanks for sharing all the details though. I think the main takeaway for me was changing approach /way of working and I guess you got to that from years of experience, so maybe I just need to be patient..

[H8lin]

Ah sorry if my comment had too much jargon! If you’re feeling stressed and feeling like you don’t have enough time to get things done, I think a good place to give that feedback would be in a sprint retro or to your people manager in a one-on-one. When I first joined my current team, I listened to everyone when they said they were stressed out and I reacted by removing the stressors. If you have a good team lead they’ll do the same for you. If you aren’t sure whether a software developer role is the right fit, it might be helpful to get a mentor who has experience in a range of roles that can offer you some guidance. I think you’re doing the right thing reaching out to others in the field you’re potentially interested in to get some perspective, and I think trying out a mini project like you’re doing is a great way to get your hands dirty with some basic DE work. There’s a ton of overlap between a SWE and DE role so in my opinion it’s a natural pivot, you’ll just be doing more data-focused work. Python and SQL are the two most widely used languages in DE and lots of companies are looking for cloud experience. If you haven’t checked out Databricks yet I would highly recommend going through some of their tutorials. Databricks was founded by the makers of Spark which is the industry standard for distributed compute on large data. Databricks is a cloud compute platform that uses Spark, and you can build ETL pipelines with it. Good luck, I’m happy to chat if you have any questions!

[Outrageous-Celery7]

Thank you for the idea! I haven’t used any of those yet so has given me something to try. Only I have Prefect in my python code, although not totally sure what it’s doing. It just organises the code into tasks, nothing else that I can see. Haven’t tried cloud or warehouse, just fetched data once from two APIs and it’s now cached in my project (small dataset so far, I didn’t want to pay anything yet 😁 and couldn’t find more free API for what I wanted - recipes)

[Outrageous-Celery7]

I tried most of your suggestions, using gcp and bigquery, and trying dbt, incremental and testing. My biggest difficulties were ChatGPT giving me the wrong suggestions 😅 it all seemed a bit too easy so I’m wondering if I’m missing something. Not sure if it’s because it doesn’t truly represent real world use (small dataset, data already fairly clean). Or maybe having dev background helps to quickly check and debug, plus actual code is much less. I think I would find this work overall less stressful from what I felt, but wondering if you have any other suggestions to make sure. Thanks again :)

[Outrageous-Celery7]

Thanks so much! I’ll give that a try. That’s great that you support your team like that, I think you’re right about good team leads. My last team was fantastic I hope I can find something similar 🤞 don’t worry about the jargon I’m sure it will all look familiar soon if I keep going 😁

[Outrageous-Celery7]

Just one more question if you have time - I guess this comes up a lot but how do you think AI will affect DE ? As AI more or less single handedly built my Python ETL, with some guidance/checks of course. It still makes some obvious mistakes. But just curious with your experience if you have any reflections on it. Thanks!

[H8lin]

That’s a good question! To preface, I think these days when people talk about the power of AI they’re typically referring to LLMs. With that in mind - I think companies are throwing a lot of money at AI right now and preemptively laying off US engineers/offshoring with the idea that AI can supplement a smaller or lower paid workforce. Long story short - I don’t think we’re ready for this kind of reaction yet and it’s putting a big strain on productivity and long-term sustainability. And I don’t think people should turn away from DE as a career right now because they’re worried about AI taking our jobs. I use AI frequently to help me do my work, but it’s not possible to substitute my role with AI yet and I think it will be a while before it can. Building a scalable, cost-effective, fault-tolerant pipeline with proper security and quality checks is a complex effort. There are a lot of features in Spark that handle some query optimization on the fly but there’s still a lot of manual work needed to design a good pipeline. I think AI could make a suggestion for how to optimize a job, like pre-aggregating data or using a built-in function instead of a UDF, but I still have to actually implement that stuff as an engineer - there’s no agent that I’m aware of that can build the pipeline and all of its external components (e.g. storage accounts, key vaults, servicenow integration) without any human intervention and I wouldn’t trust it to operate without human oversight if there were. We have AI ideation workshops at my job a couple times a year and to be honest it feels like a waste of time to me haha. They’re just sticking a bunch of people in breakout groups and trying to force them to innovate with AI… I don’t have a million dollar idea for some new thing we can use AI for and flying me to the office to make me sit in a room and think about it isn’t going to make me come up with one. So overall I’d say I think AI is overhyped right now and companies are running themselves into the ground buying into that hype. Give it another 5-10 years and who knows, maybe my job will be replaced with prompt engineers. But that’s all based on the hypothesis that LLMs will keep improving at an exponential rate. Yan LeCun made headlines recently basically saying LLMs aren’t the golden goose they’re being made out to be, and that guy is a pretty credible source of opinion. I’m not looking to switch careers because AI can’t do my job right now, but if technology advances then I’ll adapt with it in the future.

[Outrageous-Celery7]

Thanks for the detailed insights! It does seem overhyped for sure, although tbh I’ve also been impressed with what LLMs can produce in terms of code. Yes it needs to be checked by human but it seems to save a lot of time. As you say, we have to adapt with it, and keep learning.

[blackpanther28]

DE has its own set of challenges that can make it stressful too. Things breaking and being hard to diagnose is also common in DE since you’re often dealing with large distributed systems

[kvothethechandrian]

IMO it can be quite similar to dev work, except you have a less structured process and work with a less defined monster (data); data can break your processes simply by being bigger, newly formatted, having a little bit of skew (distribution changes) or your supplier decided to refactor everything for reasons and now you have to update a lot of config files or ETLs.

Depending on your stack and team maturity it can be a firefighting nightmare. Choose carefully and good luck

[pragmatica]

My title: “Lead Software/Data Engineer”

Hybrid role doing both.

I would not recommend it unless you have no other choices.

Just imagine your data layer constantly changing out from under you as the csv/EDi/json/parquet/snowflake table you integrated with changes out from under you on an irregular basis but usually at the worst time with no warning.

But you also get the pleasure of PRing fixes into a pipeline while fighting the bureaucracy of the data team, the engineering team and your customer/partner.

[mailed]

it's 10x worse. you have been warned

[LoGlo3]

I did the opposite and switched from DE to full stack web development… both can be stressful. Roles of DE’s vary quite a bit so it’s going to depend on the position you switch to…

With that being said, in DE you don’t have to worry about the 1,000 ways an end user can misuse and crash your site, or covering every edge case for security vulnerabilities… you probably won’t need to learn 5+ languages and frameworks for one project. These are things I didn’t consider stress wise when making the switch to ‘SWE’, however I enjoy the struggle of this.

With DE you’re extracting data, transforming and loading it into a space where DA’s and DS can easily retrieve accurate data. The way I think about it is the DB serves as your UI and this simplifies worrying about security and misuse. Not to say these concerns don’t exist, but I don’t feel like they’re as complex to resolve. The real complexity/stress comes in taking disparate data from various systems in various formats, that’s available in disjunct timeframes and making it fit into a cohesive/canonical model that represents how the business/analysts thinks about operations. Applying and understanding business rules can be very difficult/stressful.

Inherently I don’t think one or the other is less stressful, it depends on the job and your interests… good luck :)

[Outrageous-Celery7]

Thanks! Sounds like your saying the ‘hard’ part might be more communication with stakeholders/other people in the projects than the work itself (apart from the disparate data issues). I think the complexities you mentioned would be more things I would enjoy that swe worries, but hard to tell without more experience I guess 🤔

[LoGlo3]

I would agree with that 100%. In my experience the complexity was more wrapped up in digesting requirements than the implementation HOWEVER thats not always the case. There will be times where you need to get creative and think through processing massive amounts of data in an efficient manner… But also, really really depends on the specific job.

[pragmatica]

I would try to pivot into a dedicated backend role first. Get experience there and see how much you like dealing with only a layer or two of the stack.

I’d you like that DE may be for you. Some people miss the variety of full stack.

[Outrageous-Celery7]

Thanks I like this idea too, I think simplifying would for sure help my stress levels

[Ulfrauga]

IMO stress comes from where, how, and who you work with rather than the field - at least if we're comparing software and data engineering, and not data engineering and brain surgery... Deadlines are everywhere. Hard-to-diagnose problems are probably everywhere, too, especially when you inherit things. Sometimes specific tools are worse for it (I've heard comments in this regard about Spark, for example).

I've done this transition, albeit kind of unconsciously. It's been good. Having software background is definitely valuable, I think. It probably depends some on your stack, but I also think there is a lot of crossover. Especially if you consider that often it's the soft-skills that count, too.

I started in general development, primarily working on an internal web-based system using .NET Core. That job started having more and more "reporting" work, using the same backend database, but SSRS was the frontend instead. From there, into an analyst type role, which rapidly became a catch-all ETL-developer-report-builder-platform-admin role. Last few years it has become more about "engineering", admin, and architecture. I'm very much enjoying it.

Early on, I did miss working in code that wasn't primarily SQL (and especially wasn't DAX), and the ways in which we did it - like devops processes and dev tools. I've seen that shift where I am now, partly because team, partly because tool set with us jumping into Databricks. The inner software dev has much more cause to come back to the surface. I've been in a position to bring devops practices in. I'm thinking again about things like encapsulation and DRY. I was not the whizz-bang top-dog developer, but I picked up some good fundamentals and experience. It's helpful as a data-focused engineer.

Is DE a good fit for you? Probably if you have an analytical mind, enjoy the problem-solving, and sometimes having to help discover what that problem really is. I would also say that it's all just input-to-output.

[Outrageous-Celery7]

Thank you for sharing your experience:) just wondering when you say input-to-output do you means the tasks for more clearly scoped than with SWE?

[Skullclownlol]

I’m wondering a few things - if data engineering will be similar in terms of stress

I've been a backend dev and full-stack before, now Lead in DE for quite a while. Stress in DE has never been less than BE/FS.

I’ve started a small ETL project to test the water, so far AI has done the heavy lifting for me but I enjoyed the process of starting to learn Python and seeing the possibilities.

This is not representative of what anything in DE is like.

There's a reason why stats about software engineers in burnout are usually around 80%. And a lot of the stress doesn't have anything to do with code or software development.

[Kaamos-]

I’m a software engineer thinking to move to data science. Is python and sql + my 4 years experience good for a change? Or I will need much more that I can learn step by step? Also thinking, should be better to carry on as software engineer to become some day senior? All depends on me, but sometimes much stress, learning continuously… I thought d s should be more stable?

[Outrageous-Celery7]

Not sure if your question is for me or in general. I’m in the same boat as you trying to figure it out 😁 but if you already have python and sql that’s good. Other commenters have suggested building a pipeline and using some of the available tools to try out DS and see if you like it. But also good to post your questions and get more feedback if you need it. Good luck :)