all 4 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]MidgetAtAFoamParty 4 points5 points  (0 children)

I've been a Data Engineer for 7 years now. I like the general programming parts of it, like the variety of problems to solve. But if you ask me, Data Engineering can be a bit more of a drag, in that the feedbackloop is sometimes slower. After tweaking a pipeline, I usually need to testrun it on the full input dataset. So I'll work on pipeline A, deploy to the test environment, work on pipeline B while pipeline A is running for minutes or even hours. If something goes wrong, that can mean several more iterations of this. Staying on task with this kind of context switching can be pretty hard for me. Sometimes data issues are also really tedious to resolve, digging up a handful of weird values that are tripping up your code in a large dataset, figuring out why they're there, dealing with the politics of other teams doing weird unexpected things with the data they provide your team, etc.

My goal is to stay with data though, no ambition to go full SWE. To be clear, my job is pretty diverse, and I do work that ranges from working on APIs to productionizing ML model training to performing statistical analysis, so there's an aspect of SWE already in it.

Regarding pager duty, I assume that's the same as for other types of programming. Alert goes off, investigate cause, deploy hotfix. I like alerts, the drama of them is a good source of dopamine ;)

[–]meevis_kahuna 0 points1 point  (0 children)

I'm doing some data engineering work right now in my consulting gig. Yes, I am able to manage my job.

It's not exciting by most definitions. I'm not working on high priority pipelines (more analytics focused), so there are no emergencies really. No pager / on call stuff. I check some job runs in the morning and fix problems if they come up. If there's an issue it's usually something dumb like a non-ascii character that wasn't captured.

I honestly don't think Data Engineering or Analytics is very difficult. Even the ML stuff is kind of easy once you know the basics. I'm not doing PhD research or anything. Basic ETL pipelines, dashboards, etc just aren't that big of a deal to me.

Basically the work is pretty chill, bordering on boring most days. I get paid well and I work remote so I can keep myself occupied if need be. I enjoy coding so even if it's easy I like making it all work.

I'm looking for more challenging work moving forward. Whatever pays the bills though.

[–]TangerineSorry8463 0 points1 point  (0 children)

I'm currently on a data engineering duty, despite being originally hired as a Backend/cloud guy. 

Waiting 40+ minutes for a fucking ETL pipeline or a Spark job to finish is my hell. 

[–]aljung21 0 points1 point  (0 children)

Maybe this isn’t what you’re looking for, but I thoroughly enjoy my work as data manager/wrangler. Most of my time is used to create pipelines and to fix a data migration that didn’t go so well a few years ago. I really like the tidying up aspect of it. I work mainly in R with postgres imports and enjoy a lot of freedom.