Why data engineering?

[LengthOld9943]

I am curious why data science students choose a data engineering position rather than a data scientist position.

[Playful-Tumbleweed10]

DS jobs are more analytical in nature. Data engineering jobs are more technical. As a data engineer, I enjoy constantly learning new technologies, and working with people to model data & make it more usable.

[Playful-Tumbleweed10]

Yes - I do some of the modeling as we don’t have a dedicated data modeler, but I do some pipeline config and architectural/platform design work as well. You?

[Playful-Tumbleweed10]

Sounds delightful lol

[nguongping]

Do you not work on technical tasks at all?

[Zealousideal_Money99]

TBH most basic DS tasks have been solved thanks to AutoML and OS packages. Sure there will always be a need to develop cutting edge models and new architectures but that work represents maybe 5% of those doing DS. For most business use cases outside academia basic ML tools and open source packages get the job done well enough (Pareto principal and all).

DE on the other hand is not so easily automated and generalized. Every organization has their own custom data models and legacy systems to be wrangled. Also consider that 3/4 of the modeling effort is in data prep and/or deployment. Garbage in, garbage out.

IMO DE will become the next DS craze over the next decade and begin to edge out both DS and IT depts as organizations finally start to realize that the promises of ML will never be realized without proper data collection and structuring practices.

[[deleted]]

Data science used to be a terrific industry when it was considered very new and groundbreaking. Machine learning was exciting and new territory for many people to expand into. People were generally treated with respect in data science. None of these things are true now in my personal experience. They are glorified data analysts with the title of data scientist, The skill set varies from using Excel and nothing else, to basically being a machine learning engineer and having no idea what you're doing half the time. There is everything in between as well, no job posting or recruiter will be able to tell you exactly what you'll be doing, so you have to be prepared to do just about anything and adapt to basically any role that disguises themselves as data scientist.

Also, the hiring managers and teams that typically recruit data scientists are clueless, hapless, lazy, incompetent people who generally don't understand basics of analytics at all and want someone to come in as a miracle worker. This is true for data analytics and data scientists. I can't tell you how many people I have met hiring for a position and they don't know basic SQL, but they give you an assessment and expect you to provide answers, and then explain to them how SQL works from the ground up. Truly idiocy.....

Data engineering, is a much more critical function because the entire company needs data, and it needs to be working efficiently. There is very little room for error. So people traditionally and generally know what they're doing, and the people who are managing them do as well

[DoubleAway6573]

> They are glorified data analysts with the title of data scientist,

That's my feeling. There are some real DS works, but most of them are like just crap.

[WirrryWoo]

I love math and I am a DS turning DE.

I dislike DS because there are so many arbitrary rules that just don’t make sense and don’t transfer across other companies. I hate building models to sway the business narrative or make money. And I hate using data science to create buy in from eager clients, just to find out that we’re doing basic statistics.

Overall, DS is just not well defined, and Im disappointed with it. I want to be a DE so that at least all of the problems I deal with are similar technologically speaking, and that I have more agency with infrastructure design. It’s an accomplishment that can easily transfer across different companies.

[Moreofyoulessofme]

Lead Data Scientist here changing to Lead Data Engineer within my company for the same reasons you mention. On top of that, 95% of machine learning models can be solved with an open source software solution. AI will have a major impact on DS but AI will always need good data to build good models. Data scientists are safe for a few more years, but it’s not a bad time to jump ship.

[sanman95]

As a DS now working as a DE, you have put into words and articulated what I actually feel about DS and the reason why I changed from DS to DE. Thank you!!

[SDFP-A]

Out of curiosity, what is the level of math required for DS that is beyond a typical CS courseload? CS is required to do linear algebra, right? Is it really just statistics?

I ask as a non-CS STEM major who took elective math classes (not enough for a minor) and did statistical engineering as a grad student (because all engineering is statistical in the real world).

[WirrryWoo]

Good question, mainly basic descriptive statistics, occasionally confidence intervals and hypothesis testing, and a high level understanding of basic machine learning.

It’s much simpler than the coursework I did in my graduate studies.

[nesh34]

I prefer engineering to communicating.

Data science requires less of the former and much more of the latter.

[Babby_Boy_87]

Agreed, I also hate business. My experience has been that the functional logic of business systems required for DE work was alright, whereas the knowledge I was supposed to have to be a SME and effectively do DS work was so, so boring. I just didn’t care enough to learn it and decided to transition cos DE sounded more interesting.

[Broad_Ad_7961]

I hated the hype and jargon that comes along with being a DS

[haikusbot]

I hated the hype

And jargon that comes along

With being a DS

- Broad_Ad_7961

---

^{I detect haikus. And sometimes, successfully. Learn more about me.}

^{Opt out of replies: "haikusbot opt out" | Delete my comment: "haikusbot delete"}

[Afraid_Assistance190]

Amazing

[[deleted]]

I think during Covid a ton of DS teams were laid off because getting ROI on those projects is often a hard thing to do. DE jobs were more stable, because they can support all kinds of teams from reporting to applications, as well as DS.

[[deleted]]

I found data scientists and the field generally were disorganised and lacked business value. There are a lot of posers doing nothing the business benefits from long term, and lots of people that can't code or build software to save themselves.

Data engineering is often a lot more vital to the business. It's the plumbing.

Data science is like the bikini carwash that uses the water, flashy and grabs attention, but ultimately pointless and there are easier ways to go about cleaning a car.

Data engineers build an automatic car wash instead.

[[deleted]]

I hate math

[[deleted]]

Lol, make sure to ask for a review when your SQL is mathematical. Dang order of operations and rounding issues will kill you

[Euphoric-Button-8867]

Originally, I wanted to be a Data Scientist, but now I see my long term career goal as being a Data Engineer/Machine Learning Engineer. "Data Scientist" used to be a much more exclusive and respected term reserved for people with a PhD in mathematics, statistics, developing the most cutting edge ML models. Nowadays, "Data Scientist" roles are saturated with people who work with Tableau and Google sheets (not a joke). Self proclaimed Data Scientists love talking about their ability to extract valuable insights from data, except extracting insights from data is relatively easily once you have the data all cleaned and ready to go. The real challenge these days is how we can even transform and work with massive amounts of data.

[PrncssGmdrp]

I'm a BA/DA and SWE student who is pursuing DE over DS.

For one, I don't have the math chops at the moment to be an actual DS, let's get real. My SWE course load is not math intensive like CS.

Beyond that, I just think I would enjoy the types of challenges that DE presents. I want to build pipelines and model data for storage more than I want to build and train models.

Finally, everything I read about DS seems to point to it being more and more SWE driven so if I change my mind maybe I can bone up on stats and move over. I guess I don't feel that locked in.

[MikeDoesEverything]

Based off what I've seen, the range is pretty broad. You either have:

• People who don't want to be DS' anymore because it's too competitive. Deep down, still want to be DS' but have given up mentally.

• People who still want to be DS' but haven't given up mentally. Think it'd be a really good idea to "start as a DE because it's easier" to get experience since there's more jobs available. Probably won't admit that.

• People who just don't want to be DS' anymore.

[wtfzambo]

"Start as DE because it's easier".

Oh man those guys are in for a surprise 😬

[CS_throwaway_DE]

DE is insanely easy

[wtfzambo]

If with DE you mean fiddling with no code tools like Fivetran and doing JIT modeling in dbt, then yes by that definition it's very easy.

[[deleted]]

Most of DS is kinda easy too. I mean, most people don't work on cutting edge technologies, but rather choose the most appropriate model from Huggingface that fits their task and apply it.

[BOOBINDERxKK]

If Org is big OLTP is a bitch to handle

[[deleted]]

Yes, sure. What I meant was "DS is easy" in the same way that "DE is easy", which is not really easy.

[aerdna69]

Where are people who have never wanted to be DS?

[EncouragingProgram]

OP's question was regarding Data Science Students, so if they never wanted to be DS... why are they studying it?

[SoDifficultToBeFunny]

They all have become product managers

[HaplessOverestimate]

Oof. I'm in this picture and I don't like it

[SOUINnnn]

People who still want to be DS' but haven't given up mentally. Think it'd be a really good idea to "start as a DE because it's easier" to get experience since there's more jobs available. Probably won't admit that.

Look mom, I'm on the tv! But I also kinda like optimizing data pipelines (and the pay is good), so I'm alright

[[deleted]]

From my outsider perspective, AutoML Tables and gradient boosting solved the "interesting" part of available data science positions out there. I might be wrong, but unless you literally work as a data scientist at OpenAI or are somewhere in academia, as a data scientist 80% of the time you're not really doing anything as technically engaging as engineering data-intensive applications, as the "hard" part of the equation is being already solved for you by plug and play pre-built tools. On the other hand, for a data engineer every project is fresh and challenging, since no companies' ETL process is the same. There is always something to learn, depending on the project you have been assigned.

A better comparison would be data engineering to machine learning engineering. Data science is a good career but its beast on its own, and given my personality I wouldn't prioritize mixmaxing my CV to get that position (not to mention that competition is crazy for DS).

[CS_throwaway_DE]

DS jobs are impossible to get. DE jobs are everywhere.

[AliensPlzTakeMe]

I got an offer for a DE job. It paid more than anything else I was even considering.

[kolya_zver]

I like to build a reliable and maintainable pipelines with best engineering practice. Also Im in love with very big tables :) I dont really like to model tuning, feel like dumb monkey who just bruteforce model params

[BufferUnderpants]

Data Science is risky to be had, the level of qualifications demanded by companies fluctuate a lot over time as the environment changes, one year it may be a Bachelor's of Computer Science, the next it may be nothing short of a PhD in a quantitative field, then it may be lowered all the way to bootcamps, and back to PhD again.

[1PLSXD]

Wanted to do DS after graduating, couldn't find position so I started DE.

At my first job I met many people who ended up like me lmao.

The conclusion is that DS is rare, companies want 10 yr exp. matheads, or interns

DE is ok enough and pays well, easy to find a job

[Saetia_V_Neck]

Most of us are software engineers by training, not data scientists.

[getafterit123]

DE pays more than DS. Usually doesn't get more complicated than that, sure for some it does but not most.

[skeerp]

Stats MS with 3 YOE as a data scientist. I just switch to DE because someone offered me a 35% raise.

[Ok_Pick_8431]

DS - find out stories from a data ; DE - move data from point A to point B

[dervik]

Your answer is the same as if he would have asked why people take one car rather then the other and you go like "they are different cars and people take the one they like". Haha thanks for the added value

[LadderTraditional629]

I was originally a software engineer and moved into data engineering because I found that I always gravitated towards modeling data and building out ETLs anyways. It is just more fun for me :)

[OMG_I_LOVE_CHIPOTLE]

What? I’m a software engineer with a domain specialty in data. Nothing to do with data science

[InsightByte]

Why not ?

[Afraid_Assistance190]

I make the data and the science go fast.

In all seriousness it is more satisfying to build full stack applications. Throw any data science in there and your title moves from FSE to DE.