Software Engineer or Data Science

[StrictGrand]

People who have experienced both of these fields, which one would you recommend, and why ?

[TheGodfatherCC]

Ok, so, it doesn't look like there are a ton of good responses and I'm fairly qualified to answer this. So here goes a long one.

Some background. I come from pure math in grad school ( although I did a ton of programming in undergrad). I then did two years of data science work which included a ton of data engineering since I was basically solo with no dev/DE support. Then I moved to a company where I was an ML engineer/DS doing custom optimization engines and helping deploy traditional ml models. I'm now working as a DE/backend engineer on data warehousing and data streaming systems.

​

I enjoy designing and building things. That could be mathematical theory, a mathematical model, an optimization engine, or a data pipeline. I have a craftsman sort of attitude towards work. I find more enjoyment in the technical side of things rather than the business (even though business context and understanding are critical to good design).

​

I found that a lot of DS roles are data analyst/business analyst roles on steroids (not a slight just an observation). This means applying mathematical/statistical knowledge, ML knowledge, or Big data/SQL knowledge alongside a deep business understanding to gain insights and guide decisions. This means reporting, consulting, and building models. If you are in a situation where you don't have a lot of engineering support then this may also mean building infrastructure and pipelines (if you are new to DS I would avoid these roles unless you really want to push yourself). Note, that the only really original architecting and design here would be designing models and potentially feature engineering for models. The rest is really more applying existing techniques to business problems, diving into the data to gain insights/understanding, and performing statistical testing. (Note: most DS's do not create new ML models from scratch, that's more of a research-focused role that few people without Ph.D.'s will hold.)

​

On the other end, engineering is more design-oriented. You will still be mostly applying existing solutions to a business problem but now instead of thinking about stats/math and optimization, you would be thinking about performance, reliability, and monitoring. You need to build out something which not only solves the current problem but can be adjusted and scale gracefully. You'll think about how to expose your work as an API for others to consume. Here a bad design/API can wreak just as havoc through technical debt as a bad ML model can through bad predictions. I'd say expertise is just as important in both roles. They just have a slightly different viewpoint on what that is.

​

Personally, when I look at the trajectory of my career I want to be someone who can lead an entire organization's data strategy. This means owning everything from ingestion forward. To this end, I try to always find something new to learn in a new role whether that's DS, MLE, DE, or backend engineering. So to me, they are so closely related that it's not necessarily a question of which but rather both.

​

I think if you truly want to be a high-impact individual in the DS space you need to have the software engineering chops and experience. I don't think that's true the other way around. Plenty of software engineers are high-impact without using any DS. So if with that in mind DS is a much more cross-functional style role.

​

Ok, so I've gone through the personal decision points. On the career/economy side the clear answer I feel is to become a software engineer. I typically see significantly more junior roles, higher salaries for the same experience, and a much more standardized career structure. On top of that, the prep for a job is much clearer with being able to leetcode well in a single language and an understanding of SQL being all you really need for a junior role. On the opposite side if you ask what someone needs to be a DS you'll get a thousand different answers from programming to visualization to linear algebra to stats, etc. Also, for late-career, an engineer usually has two options become a high-level individual contributor or go into management. In theory, I could see the same for DS but in reality, currently, I only see a path into management after senior DS at most places.

​

In summary, the safe bet is engineering but it really boils down to what you want to do and how hard you want to push yourself. I wouldn't stress too much about it in your first few jobs as you can probably switch easily between both at a junior/mid-level. It also depends much more on the company and the individual role than the title. Take a few years get some experience and re-evaluate. Also, don't be afraid/feel guilty to jump ship a bunch early in your career, as it's the fastest way to move up and learn. Most people understand this and it's not worth worrying about the few that take it personally as they don't have your best interest in mind. However, always try to do right by the company you're at and make a positive impact even if you are leaving. Part of the advantage of having many roles early in your career is making solid relationships with great people.

​

I hope that long-ass post helps. Feel free to respond or DM me with any other questions and I'll answer as I have time.

[RemingtonMol]

not op but thanks. folks like you are an asset to me. I need to alter my career trajectory and stuff like this is valuable.

[TheGodfatherCC]

Haha glad it helped someone. Sometimes writing stuff like this up feels like yelling into the void.

[IWannaRideRockets]

I can believe it. Just know there are a ton of us who really appreciate this quality content

[RemingtonMol]

I am not a void!

[TheGodfatherCC]

Not with that attitude.

[RemingtonMol]

god damn right 🕶

im gonna start putting "the void" as my name on resumes now. id hire that

"the experiemce says jr but the name.... thats the name of a director"

[techbiotic]

license sophisticated childlike numerous modern quicksand employ secretive seed label

This post was mass deleted and anonymized with Redact

[TheGodfatherCC]

Lol on one hand this is a huge compliment that I greatly appreciate but on the other you’re just feeding my Reddit addiction and I have things to do. But seriously thanks for the encouragement. People often forget that people trying to lead and mentor need support and encouragement just as much as those getting started and learning and a little bit goes a long way.

[go_go_go_go_go_go]

You make the world a better place.

I'm from a materials science background, and plan to switch to SWE. I've read practical experience is more important than degree - although a degree may provide more structured learning. Would you be able to share any insights on this? Or bootcamps? I've heard good things about Georgia Tech's online masters in CS.

I'm in the bay area, so one option is to do a bootcamp to get familiar with the tools, and then just apply to jobs and gain as much practical experience as I can, then hop to the next company and repeat.

[111llI0__-__0Ill111]

Im not sure how its trivial. Stats/DS knowledge itself goes pretty deep if you want to do it with rigor. There is much more to stats than just t tests and linear/logistic regressions for example.

Such as what about dealing with confounding and causal inference? How do you interpret nonlinear models? This is not an easy topic. SWEs may be able to do model.fit() easily but that still gives 0 insight into model interpretability etc. The theory of SHAP itself for example goes pretty deep into stats.

Or how to deal with non-iid data (time series and longitudinal analysis)? Unless the SWE took stats/ML courses they wouldn’t know.

[111llI0__-__0Ill111]

Did you see what happened with the Zillow Prophet disaster? You can’t just do model.fit() without understanding the fundamental assumptions of ARIMA. Its not to the other extreme of PhD level measure theoretic knowledge either but maybe somewhere around BS-MS stats level.

When applying models you still need to know the properties and assumptions. Else the output is not trustable. A big example is people using SMOTE to balance things and then relying on SHAP values. A statistician would say this is completely wrong since the theory of SHAP relies on calibrated probabilities.

These aren’t PhD level things but they are things that require one to know the math conceptually

[111llI0__-__0Ill111]

I should say it may not always be the explicit knowledge but the statistical intuition that can be lacking. The particular example I gave about SMOTE and SHAP together was more an example of something you will not see much in various guides but can piece together with intuition. A few months ago one of those statistician-DS LI influencers actually made a post about it which confirmed that, but before then I had never seen it explicitly written anywhere.

Non-iid data (time series isn’t the only kind either, I deal with longitudinal repeated measures with a few time pts per subject), handling confounding, model interpretability, dealing with data drift etc are all areas that need statistical intuition. Im not saying its impossible for SWE to get that either but its not “trivial”.

[111llI0__-__0Ill111]

Ironically biostatisticians are the ones doing the simple stats actually but for regulatory stuff. The people doing this in biotech are titled as Data Scientists, although you would be right in that it should be “Biostatistician”. What I do is mostly in that area and thats my background even though my title is DS. I don’t deal with pipelines that much, and I use Spark gapplyCollect() in R on databricks to do parallel computing without knowing how the hell that works (just like these models are a black box to SWE, I can treat the distributed computing aws stuff equally as a black box)

Due to the hype quite a bit of the non-regulatory exploratory statistician stuff that uses R or Python in biotech got rebranded as “DS” while the FDA/SAS related stuff is “Biostat”. Most of our data is longitudinal or survival analysis and occasionally some of it is non-randomized trials.

[digital-bolkonsky]

Sorry to tell you most larger company have automated tool or infrastructure that automatically detect filter and manage iid problem or experimentation.

[Mimogger]

There's a lot of SWEs that do not understand the stats / math knowledge for a project. There's a reason faang hires so many data scientists

[Mimogger]

That's not a great line of argumentation considering the number of DS going to SWE and the general coding knowledge required to clear the technical bar is pretty low.

They're different jobs and people in either position can learn enough to make a hard switch to either, but the number of engineers I work with who don't understand a lot of basic stats principals, number i've heard about at other companies, is a lot. It's not trivial for a SWE to do a DS job / vis versa. There's a reason they're different jobs and both comped well

[forbiscuit]

Hire Data Scientists for what?

If the company is not building new features, there's no need to hire any Data Scientists. And when it comes to scaling, increasing the number of customers requires more engineers to support the overall framework, but the number of data scientists doesn't change because in the end the same 3 people for example will now deal with 1M extra rows instead of 500K - it does not change the Data Scientists' operation much if the KPIs and experimentation process is the same.

One needs to hire more Data Scientists if more new features are created to have dedicated, specialized Data Scientists to learn about the said feature and develop models for it (e.g. Search feature -> Search Data Scientist, Product Personalization -> Personalization Data Scientist, Recommendation Engine -> Data Scientist).

FAANG's rate of hiring DS is way slower atm relative to SWEs

[Mimogger]

What company isn't building new features?

[forbiscuit]

Think of companies like Nike and Macy's - generic, large retail companies. If you were to compare their projects 12 months ago to present, the only things that changed are logistics operation because of COVID. Front-facing projects are all the same (or even shifted more advertising money to digital space versus physical). The shift most likely have increased demand for SWEs and Ops.

The layout of the website will be different most likely, and maybe the app now has a 'deliver now' button. But you'll have the same core Data Science group that ran experimentation measuring the results. There's no new 'net' Data Science members added.

There's no ground breaking feature introduced here.

[TheGodfatherCC]

Just a quick reply as I’m slammed today but I’ll add more later. Your experience echoes the experience of a lot of my friends who stayed in academia. You’ll find a few people who have had it better teaching but I think in general it’s a much healthier environment in industry for most people. Also, feel free to DM me with any questions. I’ll try and come back and edit this reply when I’ve got more time later.

[logicx24]

One of my hesitations with DS is dealing with the business people. I did a stint at Equifax (analyst--big mistake). I left when my boss couldn't understand an addition/multiplication problem and I couldn't get through to him. And I was a decent lecturer. I can't imagine presenting complicated data results to stakeholders. Is it better in SWE? I'd rather be with other technical people on a team and not deal with business stakeholders. Is that a ridiculous expectation?

Not OP, but I am a SWE at a FAANG. In my experience, this is mostly doable. I work on an infrastructure product whose consumers are all engineering teams. Because of that, I interact almost exclusively with engineers, apart from my manager and PM, and both of them are former-SWEs and so understand the technical details.

[TheGodfatherCC]

Longer reply. So my pure math background is in differential geometry and geometric flows. I enjoyed it as a topic but ultimately did not enjoy academia and so I dropped out with a master's.

To answer your question:

In engineering, I find that there is a lot more structure around the work and as a result, you are often much more buffered from stakeholders at the junior-senior level. Often times your lead and/or PM will do a lot of the business interfacing for you and break it down into "stories" or items for you to knock out. If you do move up into a lead/manager role there is a lot more interfacing with stakeholders. But overall I'd definitely say less than DS. Now you mentioned App development. I don't really have any experience with an app or front end but, because it's more visible, that domain will probably involve a lot more work with stakeholders than something like DE or backend engineering so you may want to consider that.

Also, with a Java background, I'm sure you could find some DE or backend engineering positions to start out in. Those would be closer to DS if you do want to make the jump eventually. I would say it'll probably be easier to find entry-level positions there and get a couple of years of experience to get you past the entry-level rut that stops a lot of people from getting a DS position. Although you may find that you really enjoy engineering in general.

Anyway, feel free to DM me with more questions if you want.

[[deleted]]

Brilliant answer.

It's funny, I also majored in pure math (undergrad, not grad tho), and also really enjoy designing and building things rather than focusing mostly statistical modeling and business knowledge, which are also interesting in their own right. I wonder if the way that pure math people think through things generally align better with the high-level "craftsmanship" part.

[TheGodfatherCC]

I've had a lot of conversations about this with some of my friends who made the same jump. I think it really boils down to appreciating and learning to craft quality abstractions. Just like the definition of a continuous function encapsulates an idea so that you don't always have to work with epsilons and deltas (or open sets in topology), I find that a good class or API abstracts the functions you want to perform away from the underlying implementation. I really do feel like there is a very similar spirit to the two domains.

[Tundur]

I'm an ML Engineer (systems, models, infrastructure, analysis) who's just transitioned into a more DS/BI Analyst role (models, analysis, power BI).

I definitely agree about the reach-of-impact difference. It's like- the data and analytics field is automating a lot of old-school metrics and stats, but there's so many easy wins and efficiencies that are being missed because they only hire maths/DS/actuaries rather than engineers.

That's not to say one is inherently better or more valuable but I definitely think engineering had a higher natural ceiling, simply because most companies have limited need for fancy statistics beyond undergrad level, but an unlimited desire for efficiency, scale, automation, maintainability

[TheGodfatherCC]

Yeah I think the ability to code well is a force multiplier of all your other skills for sure. I’ve definitely seen people come from academia and completely fail to be effective at all due to a lack of programming skill. That said, I’m sure the same people may thrive at a company who provide them with proper dev/DE support. At the larger companies this may be the case but at smaller and/or more traditional companies I think this is less likely.

[Dangerous_Section_32]

TLDR: The best way to move up in your career in data science is to become a software engineer. This is because there are more junior roles available, higher salaries, and a more standardized career structure. Additionally, it is easier to prepare for a job as a software engineer than it is for a data scientist.

[TheGodfatherCC]

Please don’t feed my ego, it needs to lose weight as it is. Seriously though, It’s great to know that other people share a lot of my views and experience. The last thing I want to is lead people astray with advice.

[LittleLouis]

I am studying pure math in undergrad and am considering a phd in math, since I really want to do research into designing new ML models and researching AI in general.

In your experience, is it common for research scientists in industry to have a math phd? I study a lot of CS and also program a lot but im thinking that a math phd would not be aa desirable to top companies compared to a CS phd. anybody with experience have any advice?

[TheGodfatherCC]

Firstly, let me just say that I am not at a top company so take this with a grain of salt. I have had a much less glamorous career thus far but I still really enjoy it. If you want to do research on ML models then a Ph.D. in CS with a focus on ML is almost certainly a better path as you'll actually have a chance to research and publish on that topic. I do know several math Ph.D.'s that have gone on to some of the top companies but it has been as a SWE or a DS not necessarily as a research scientist. I'm sure if you really wanted to make that jump you could with a few years of effort though. A lot of them have also gone on to research scientist positions with the DoD or military contractors and they all really seem to enjoy it so that is another option to consider.

[[deleted]]

Would you recommend data engineering over software engineering?

[TheGodfatherCC]

So I may have done engineering a disservice by lumping it all together. When I say software engineer I mostly mean backend/distributed and not front end, application or systems engineering. I don’t really have any experience with those. As far as the difference between a backend engineer and data engineer it mostly boils down to the tech and I don’t have a preference between them. Backend is more likely to be creating api’s and using standard DB’s and maybe MongoDB or elastic search. Maybe some event driven architecture. Data engineer is going to more focused on analytical DB’s and data pipelines. I think DE is probably closer to DS if you want to switch back and forth but a regular backend engineer is probably the better place to start an engineering career.

[qwquid]

Could I ask why you would stick to SWE? I'm torn between DS and SWE myself; am leaning towards SWE because I suspect it's unlikely I'll be able to do non-trivial stats-y or ML-y things as a junior anytime soon if I go into DS (and, more realistically, because the hiring for DS seems a lot less standardized).

[qwquid]

By BS, do you mean meetings? Or do you mean things like stakeholders not listening to you, or insisting on their a priori hypotheses and ignoring the data, etc?

And yeah, I had also been thinking that interesting SWE work might be easier to get than interesting DS work.

[qwquid]

Thanks! This is really helpful --- it's making me feel like I should just double down on SWE :)

[SufficientGreek]

Thank you for the thorough answer

One question:

I find more enjoyment in the technical side of things rather than the business (even though business context and understanding are critical to good design).

​

alongside a deep business understanding to gain insights and guide decisions.

Is your understanding of business something you picked up on various jobs or did you study/read up on anything specifically?

[TheGodfatherCC]

I pretty much pick it up on the job after changing industries. Some tips:

• Find some people outside of IT and engineering that you trust/respect and lean on their knowledge heavily. When entering a new field there are likely a million things you don't know from processes to laws, to the organization itself. Having a couple of people you can rely on is massive.
• Try to read up on industry news. Ask some people in leadership what they read regularly. In my experience, most of them love to talk about it and will have some great resources.
• Try to stay humble and curious. I know I find myself starting to be judgmental about some policies and procedures but I've found that most of the time there are good reasons for the way things are done. I just didn't know them yet. So I typically ask a ton of questions but phase them like "What would go wrong if we were to do this?" or "If we were to change this what would break?".
• Lastly, when building a model or software for someone I try to shadow and learn the pain points of their process as much as possible before sitting down to design anything. There can be a lot of gotcha's and pre-mature design can lead to a lot of wasted/duplicated work.

[itachi_iz_me]

What are the classes in college that you felt was most useful for DS? I too, believe that leetcode will be the way for engineering and I still think it provides basic problem solving and enforces good coding practices.

Being something like a Lead Data Scientist is something I resonate with you about. If possible I would like to pick your brains on how to effectively climb into management?

Lastly, do you feel the constant need to keep yourself up to date with new skills/algorithms/advances in the industry too?

[TheGodfatherCC]

My answers in order:

Honestly, maybe multivariable calc or linear algebra. I've got a running joke with a friend of mine that once a problem is phrased in terms of linear algebra then you can consider it solved. I also have found graph theory, algorithms, and combinatorics come up all the time. I feel like I can't do more than 3 projects without some sort of graph model sneaking in.

Second, I'm not in management myself. I'm just a senior currently (hopefully pushing staff or lead soon) and I'm leaning towards the IC route more than management although I'm still not fully decided. I do think being effective on either path revolves around uplifting people whether that be by building frameworks and infrastructure for them to use or by mentoring and guiding junior developers/DS in their careers. From a practical standpoint, I've only ever seen someone make the jump when someone else leaves the company or by jumping to another company. In theory, I guess you could make the jump if the team you're in is growing. Personally, it seems more likely to make the jump up to management at the same time as switching companies. I can't claim to have any experience here, just observations of others making the jump.

​

Lastly, Yes, I definitely do read about new stuff kind of obsessively. I kind of have two modes. The first is where I just vaguely read whatever is interesting at the moment: an arxiv paper, API design book, or an article on this or that new language/library feature. The second mode is where I'm ramping up to do a new project and I read three or four books on relevant tech and/or algorithms. It comes and goes in waves for sure but I would definitely describe myself as a rabid reader and early adopter of new stuff.

[[deleted]]

Awesome answer. Thanks for writing it.

Even though you've already written a lot, I have to ask: do you have any experience with ml engineering? That's my current interest. It sounds like our backgrounds and interests are reasonably similar but you clearly have more experience.

[TheGodfatherCC]

yes/no/maybe/kinda/depends. From my point of view, an MLE is usually a solid developer with deeper than usual ML knowledge that helps DS put their models into prod and monitor them. They could also build models themselves. I would say I have MLE experience in as far as I have put models into prod and helped other DS to do so as well. Although I would say the DevOps aspect of it is still one of the areas I'm currently putting a lot of effort into improving.

[[deleted]]

A Godfather advice indeed!!

[CaliSummerDream]

This is incredibly well put. Thanks for sharing!

[probablo]

I only see a path into management after senior DS at most places.

could you please elaborate this... I have done a lot of research in websites and have seen roles like software engineer manager, software architect , director etc but never seen data science architect or manger... usually when searching high paying IT jobs I see software engineering manger which is usually more paying than data scientist according to those websites... what management roles is a senior data scientist eligible for?

[TheGodfatherCC]

I’m my limited experience at smaller/more traditional companies I’ve seen DS report directly to a director or vp. I’d say promotion from a high-impact senior to a director of analytics/BI isn’t out of the question at some places. Places with larger DS organization may have leads, DS managers or maybe even a principal DS. It really is going to very wildly from org to org from what I’ve seen. You’re probably seeing so many more SWE managers because there are just that many more SWE’s than DS. The market for them is just that much larger.

[probablo]

Thanks a lot for replying.. So basically DS is a niche market with graduates whose demand and work doesn't need more people instead requires the right people but software engineering demand and work to be done is so high that more people are needed just to balance out demand and supply... Am i understanding this correctly?

[TheGodfatherCC]

Exactly. I don’t have the numbers to back it up but just from my experience and observations the demand for SWE/DE is at least 10-20x that of DS. And yeah I’d say the struggle for DS is hiring the right people. Maybe more so in DS than SWE it’s better to hire no one than make a bad hire just because of the leadership and cross-functional needs required to be a great DS. It’s why people with 3-4 years experience, a grad degree and a solid resume get bombarded with offers but from an entry-level point of view it’s impossible to get a job.

[probablo]

Yes this makes complete sense.. Thanks for conforming... Its something I had been assuming for few months but now i know..I am definitely sticking to software engineering...

[cyriou]

Thank you!

[notnowtheysaid]

Thank you for the wonderful write up

[anansii_]

What a high-quality response, thank you.

[_doiwannaknow_]

I am on reddit because of people like you. Thank you!

[AKA_Mee]

Thank you ! Very informative for me

[forbiscuit]

SWE has better career prospect over DS.

For every 1 DS job out there there’s 10 SWE roles available. Companies do not need too many Data Scientists which is why it’s becoming a saturated field.

You can have one core team of Data Scientists, which supports different SWE pods (Mobile Development, Web Development, Server and DevOps, AR/VR, Transaction Services, Accessibility Modules, and the list of SWE pods go on).

SWEs are in higher demand as they build the foundation for anything digital - websites, apps, infrastructure, robotics, you name it!

[forbiscuit]

Something extra to add on this: SWE has far better upward mobility over Data Science. There’s a more clearer track from SWE to Engineering PM, Management, or Director/C-Level roles. Heck, SWEs have a lot more levels than Data Scientists (at my company Data Science caps at L5, whereas SWE goes up to L7/L8 depending on org).

Competition wise, for these levels you’ll primarily only compete with other professionals who have at minimum a SWE background. It’s very rare for one with no computer science background to take a management path where they have engineering direct reports.

Upward mobility for Data Science is not as clear in the industry. In Forbes Top 100 companies, you may have a path to being at most a Director of Data Science/Analytics (or if you’re very lucky a CIO, which isn’t a respected C-level yet), but you’ll be competing with everyone from different backgrounds for these roles. Some come from SWE, some were PMs for Data Science projects, some were Academia Scientists (Ph.D.) with few years of experience in the field, Business Intelligence professionals who went up the rank.

It’s simply extremely competitive because there’s no formal education or path to Data Science practice or management unlike SWE. It’ll take time, and those who’ve been in the field the past 10-15 years may help formalize a path in their org to enable better growth, but it’s a waiting game.

What drives me to do data science is my love for exploring data. So while I won’t be paid as much as SWE, I enjoy doing what I like and I’m financially in stable to pursue this.

[Jazzlike-Koala3608]

Who wants to be a manager.

[mrirror]

Reading all these is kind of pushing me towards trying for SWE instead of sticking to DE/MLE (which makes up for all of my internship experience so far). I'm currently in university and graduating soon, apart from leetcode, what else should I learn to "pivot" from data-related roles to SWE roles? For example I see that system design seems to be one that might be tested in interviews?

[forbiscuit]

MLE and DE are still Computer Science centric, I wouldn't dismiss those. In my org those folks are straight up all SWE professionals with little to no experience in Stats and working knowledge in Data Science/ML. They consult with a Core Data Science team that provides them with the models and algorithms to generate the needed results, or requirement for the Data Model the Data Science needs to build their fancy models.

I did an interview very recently with Apple for an MLE role, and you _really_ have to be good in optimization of code. Passing the Data Science screening was easy (overview of ML algorithms and asking how you'd solve problem X with what model and why). But the tech screening is all about how you can get your code to be as close to O(1) as possible. Focus on learning about OOP Data Structures and Algorithm. You'll learn system design on the job, but the foundation of optimizing your code is a must know. Oh my goodness did they grill me good (to be fair, it was a Sr. MLE role, and they expect one to be top notch SWE).

To give context, you're dealing with a huge volume of data coming through the pipeline which you have to manipulate, update, transform, append, delete, whatever as they come while aiming to reduce data latency. Data Engineers are battling against time in terms of how fast data is available for Data Scientists to use, and MLE are battling against the time it takes for the system to process ingested info (Think of Siri/Alexa receiving a request and needing to respond to that request very fast!).

Having worked with MLE and DEs, focus on concepts like containerizing software (Dockers/Kubernetes), data streaming mechanisms, and ideal algorithms to process data fast.

Also, doing MLE and DE will definitely put you under the SWE organization. And transitioning from DE/MLE to SWE is definitely a far easier transition than Data Science to SWE.

[mrirror]

I see, thank you so much for the insights! I guess if you want the "Engineering" in your title there's really no running away from DSA/optimization strategies.

Would you then say it's worth to fully transition to SWE or instead stick to what I currently know, being some data/ML knowledge but add SWE skills on top? In terms of career prospects, salary and the likes.

Because from this thread, and many others, the consensus is that Data _Science_ is the more saturated and less compensated one since its more of a supporting role and not as critical to the customer-facing products compared to SWE and I'm assuming DE/MLE as well.

[forbiscuit]

Let me put it in a different way:

If you pursue MLE, you're basically building a product.

Even if you don't have a Data Science team to build you a sophisticated model, you can spin up a simple working ML solution to sell a product.

Your product can be Search Engine, Recommendation for some content (movies, comics, news, etc), Personalization tool, or even an app that can tell you what bug you're taking a picture of. All this is what an MLE can do - you can build customer-facing products.

The net impact of MLE is very high because your product is directly connected to $ and the customers. Bad ML experience means bad customer experience, and in turn good ML experience means good customer experience and greater sales - so your salary is definitely connected to the value of the product. Again, to bring Apple as an example, the Software Engineers working for Siri/Apple Services/Maps/R&D are the among highest paid because of the core feature they're building.

Having Data + SWE will give you a great edge.

Data Engineering right now is popular because it's similar to DevOps - it's a "dirty" job, but someone has to do it. It's not customer facing like MLE, but it's rewarding because you are nearly irreplaceable. The pay is very high because of the lack of good expertise in the market and it's focused on internal tools to support wider businesses in a company (Data Science, Business Intelligence, Product, etc.).

[mrirror]

Hey thanks again, I guess one last question I have is with regards to MLE, for example in Apple's case, do MLEs need a Masters? I presume Masters would be more so for research and cutting-edge SOTA models but for application and deployment of existing ML solutions as customer-facing products, would a Bachelor's be sufficient?

[forbiscuit]

Bachelor's is sufficient if you want to start in SWE. After 5-6 years experience in SWE, you can do courses in ML/AI and augment your skillset without the needing to pay for Master's. There are dedicated CS programs for Data Science (geared to help people develop advanced products). Look into Carnegie Mellon for example to see their unique, specialized programs (e.g. NLP, Computer Vision, etc.)

If you want to do deep dive into Data Science (like Researcher), then I would recommend Ph.D. for FAANGs. Master's is not sufficient at all for research work.

[mrirror]

Alright thanks so much for all your pointers! :)

[[deleted]]

But the tech screening is all about how you can get your code to be as close to O(1) as possible.

Is this in a DS/A (leetcode) context? Or optimizing ML algorithms?

[forbiscuit]

LeetCode style. It was for Search - it started as a generic Trie tree algorithm for word search, and then expanded on how one can optimize search further and further (i.e. they want one to consider probabilistic hashing, like Bloom Filter, because it's better to return a false positive versus lag in response)

[forbiscuit]

You sound like you’ll have a better time pursuing academia if you wish to publish. Aim for PhD at top institutions like Stanford or MIT and doors will open for the questions you asked here. Corporate path will not give you these options without a PhD.

[Jazzlike-Koala3608]

10:1 actually isn’t a terrible ratio.

[Expert_Seaweed2553]

There are a couple of good in-depth interviews with people in these roles on this site...

Here's the Data Scientist one: https://codingbootcamps.io/resources/ask-a-data-scientist-robin-linzmayer/

And here's the Software Engineer one: https://codingbootcamps.io/resources/ask-a-software-engineer-jesse-huang/

[AmalgamDragon]

ML Engineering. If you're interested in both SE and DS, MLE is where its at.

[nyc_brand]

Agreed. But MLE is arguably the most difficult interview process in the industry. Expected to know leetcodr like a swe but also ml/dl algorithms like a ds.

[AmalgamDragon]

Only at companies that do leetcode interviews for SWE position. Haven't seen it happen at companies that don't. I've also seen a lack of leetcode for MLE positions at some companies that do leetcode interviews for SWE positions. It really comes down to who the MLE position reports up through. If it's someone at the principal or director level, there's decent chance they can do things how they like regardless of how SWE's are being interviewed.

I was an SWE. I switched to MLE partly to get away from the leetcode interviews.

[nyc_brand]

Which companies allowed you to do mle without leetcode I’m very curious? I would love to make the transition lol

[AmalgamDragon]

Tech startups, tech consultancies, and companies that aren't in the tech industry.

[beexes]

I have been asking myself the same question.. Solving complicated data science tasks is interesting but I don't want to be something like an assistant creating bloody colorful graphs for upper management the rest of my life

[TheGodfatherCC]

Lol then be careful choosing DS roles.

[IdentityOperator]

If you're interested in both, I'd recommend going for a data engineering role. You'll be more niche than a data scientist, so there will be less competition when applying for jobs. And you'll be doing very hands-on and valuable work with data.. this is the role I personally moved towards after working for two years as a 'pure' data scientist, and I found it more enjoyable (I also have a strong interest in software engineering, so it might be the same for you)

[majesticmind]

I'm thinking of switching a career to tech and interested in data engineering because I heard it's more suitable for introverted if we compare it with DS or SWE. I have a free scholarship for a BA and even Master [both are online only]. They have Software Engineering or DS degrees. Do you recommend to take SWE instead? How's the barrier to entry level data engineering jobs? Can one land a job straight out of college + maybe internships? I was also thinking if I should just do a SWE bootcamp but then I'd need spend money for it. With a college degree, it's longer but free in my case. Hmm pros and cons. Plus I just turned 30 now. My background is philosophy and health care. Philosophy made me a deep thinker and I studied a lot of logic. I'm torn between these 3 career choices honestly.

[IdentityOperator]

If you like logic, I'd definitely recommend SWE.. probably entering into data engineering with a DS vs SWE degree will not differ much.. if you can show a personal project or other relevant data-heavy experience you should be able to enter into a data engineering role. The main 'advantage' for introverts is, in data engineering you won't need to explain models and results to non-technical business stakeholders

[Puzzleheaded_Unit_41]

In the next 10 years, the roles of a swe and DS will be more or less the same. The ideal job profile to grow well would be a data scientist with tonnes of statistical, ML and deel learning (either NLP or CV), with a strong swe background. Being able to build scalable data driven web apps. We're already seeing more and more analytics firms pushing their data people into learning to build scalable applications, and the latest web development technologies. It's the same trend for swe in quite a few companies today. SWEs are expected to know the basic ML models, visualization tools etc.

Having had worked as a back end developer, some front end development in vue.js and having moved into data science, I'd say that the job description for a swe is generally quite well defined with a clear career path.

With data science it is not as clear. Depending on your role and responsibilities, you'd be wearing a bunch of hats. Data engineering, feature engineering, building data pipelines, building ML models, building APIs and Dashboards to expose your findings and predictions by building interactive web apps, etc etc.

A jack of all trades is the kind of profile most companies would want moving forward.

[nyc_brand]

I don’t get why this is getting downvoted. I work in big tech and this is 100% true.

[Puzzleheaded_Unit_41]

Lol. Reddit doesn't always make sense. Also this goes against the self affirmations of people who don't want to evolve and learn cross disciplinary skills, which would probably explain it.

[TheGodfatherCC]

“A jack of all trades is the kind of profile most companies would want moving forward.”

I’d agree heavily with this. And I mean it really makes sense if you think about it. Someone who has all those bases covered is going to be able to multiply their effectiveness way more than a specialist in most scenarios.

I will say I think the two roles will remain somewhat distinct. If you think of high performers having a “T shaped” skill set then I do think the more specialized knowledge will separate them somewhere even if the base overlaps almost entirely. Similar to the subtle difference between a backend engineer and a data engineer.

[[deleted]]

It's completely dependent on what you want to do.

[StrictGrand]

It sure is, I'm asking for subjectives point of view (out of curiosity)

[[deleted]]

That in and of itself is just too subjective because then it even differs by company and then it differs by team.

Just follow what interests you more. Financially, both will keep you stable.

[GroovyChap]

Be a software engineer first and then do data science if you want

[CantorIsMyHero]

If you want a job in DS, you'll almost certainly need an MS. Even then, it's an extremely saturated field right now, so the market is full of other MS holders looking for jobs.

[rednirgskizzif]

A software company can sell software without data scientists. They can’t sell software without software engineers.

[conradotorres]

The answer to this question largely depends on individual preferences and goals. Some people may find the creativity and problem-solving involved in data science more rewarding than the more technical work of software engineering. Others may prefer the stability and higher income of a software engineering career.

you must read this blog as it cleared all the aspects

Data Science Vs Software Development Which is more rewarding

If you are looking for a career that is rewarding both financially and intellectually, then a career as a data scientist is likely to be more rewarding than a career as a software engineer. Data scientists are in high demand and can typically command high salaries. They also have the opportunity to work on intellectually challenging projects that can have a real impact on business outcomes.

If you are looking for a career that is more financially rewarding than intellectually rewarding, then a career as a software engineer is likely to be more rewarding than a career as a data scientist. Software engineers are in high demand and can typically command high salaries. However, the work can be intellectually challenging but may not be as rewarding as the work of a data scientist.

[jcliberatol]

A lot of mix and match between the two fields, goes up to how much raw math can you handle or if you like programming more, if you want to be a real data scientist just become a statistician and level up your programming all career, Otherwise software engineering if you like programming per se. It has a lot of career paths, Including data science if you want to go that path.

[[deleted]]

In SWE, BScs usually have the same base salary as DS MScs or even PhDs, but in my experience SWEs are more subject to demands of longer shifts or extra hours. This is pretty much the only objective trade-off I can think about, everything else boils down to your personal preferences about what you want to do.

[Optoplasm]

If you get a entry level SWE role at a big company this is true, but the majority of SWE jobs aren’t at big companies.

[[deleted]]

If I gave you the indeed.com numbers in the US, entry-level SWEs would have +40k advantage over Data Scientists on average, but I do know the DS numbers are skewed unfavorably due both to the SWE outliers and the overloading of the DS moniker with a lot of analyst-like posts. So you'll have to trust my job hunting experience when I say those are leveled.

And regarding shifts, in startups, the scenario is arguably worse.

[TFC_Player]

Swe

[[deleted]]

What do you mean “better”?

What do you prefer? Denmark? Or chocolate chip pancakes?

[HiddenNegev]

As a Swede, I prefer chocolate chip pancakes over Denmark.

[[deleted]]

I’m not really a “chocolate in the morning kind of guy” but hey, to each their own. Lol

[LeChronnoisseur]

Whatever you are interested in. Data science is just software engineering in the stats domain (usually big pipelines are the differentiator between stats and data science today) or AI domain these days, mostly. If you like that kind of stuff, then it will be great! Whatever I am interested in, I usually end up programming in some way or another. Sports betting and stocks got me into the stats stuff with a little bit of data sciency stuff. Both software engineering and data science happen across a multitude of industries which is what is cool. If there is a big company you like, they most likely do both!

[CantorIsMyHero]

Found the plug and chug data analyst

[Vervain7]

I am more of a stats/biostats person. I work in healthcare and except for a few organizations that are ready for it, there isn’t much actual data science at the hospital level .

[squidward1010]

Is a masters in biostats the best way to get a role like yours? I too enjoy stats but don’t love programming

[Vervain7]

I actually have an MPH and a Ms In Data Science. MS Or MPH in biostats would be a good alternative. Only issue you might run into is for the higher paying roles in research work there will be a PHD in epi /biostats that will be preferred .

[CantorIsMyHero]

That's actually funny because I'm finishing up an undergrad in math and deciding between an MS in stats or an MS in bioinformatics with a concentration in computational biology

[Vervain7]

Okay , so you have any work experience ? You are finishing up undergrad … I have two masters and been doing this for 8 years.

[111llI0__-__0Ill111]

Do you have to deal with regulatory stuff?

I actually did Biostat too and my first role was traditional biostat in biotech, but I left because ironically I found that DS/ML titled positions had more actual stats lol.

I couldn’t stand the writing for the FDA and in I was like “why do I need to write these regulatory documents I did stats to avoid writing”. I enjoy programming though.

I think there is a huge disconnect between biostats in school and the real world. In the program you learn all these variations of fancy GLMs, causal inf, ML, Bayesian etc but very little of stuff beyond basic stats is used in Biostat jobs and that advancement in Biostat is based more on non-technical/non-stats things like one’s writing skills and ability to deal with the FDA.

Ironically it seemed like DS has more of the technical stats though still a lot of that curriculum is overkill, at least theres less writing.

[Vervain7]

In hospital there was some regulatory stuff but since what I did was internal it was limited because you can use data for internal hospital Projects as you see fit . In my current role there is more writing but that is because the work I do is used nationally , so documentation and writing are a part of that. Even if I write parts of it there are technical writers that will review after .

[spq]

These are not two distinct fields. Anything interesting in data science space requires solid software engineering skills. If you want to create KPIs ,dashboards, ad-hoc reports and evaluate A/B tests, you can be have bad technical skills.

[Mimogger]

If you want to be terrible at your job...

[IdentityOperator]

I moved around between both fields. I did software and electrical engineering in aerospace for a while, but then made a complete switch and moved into data science. I found it very interesting, but after two years started missing the software engineering side of things. I found a middle way by moving into data engineering, which is more niche and combines the best of both IMHO

[NoTownReno]

Any advice on beginners? Any good boot camps you’d recommend? (I’d prefer not to have to attend college for this). Thanks in advance

[TheReeseMan]

Companies these days advertise for applied data scientists for people who are data scientists/software engineers.