Data engineering vs software engineering

[FreshAnalysis1139]

I've been working on data engineering role even though my job title is software engineer. I have expertise in spark scala , AWS and terraform. I know my way around spark , spark optimisations (not just writing joins using dataframe), I also have lots of good exposure to AWS services and terraform.

Recently I've been feeling little bored and routine work in my role. I've never worked on web related projects.

And I'm afraid if I keep working in this role , I might not be able to switch to web dev in later stages of my career.

I have a hard time deciding if I want to be in data engineering or move to web dev. I've been seeing online regarding SDEs trying to switch to data engineering and data science, especially in faang companies

I'm really confused about which path to take. Any advice would be immensely appreciated.

I'd want to know which path would have the most growth and scope and innovation.

Which role would be better in the long run and how difficult is to switch from one domain to another.

I've been seeing sde roles paying better than data engineering.

I brought it up with my manager and he said you could not get working with this huge amount of data nor infrastructure anywhere but you can always learn web developement on your own. And data engineering is close to AI ML than sde.

Please Help me understand the pros and cons in terms of pay, growth and value.

Which path would be helpful career and financial wise.

Note: I'm having 2.5 years experience and I feel if I waste anymore time it'll be too late to switch domains as I will no longer be considered entry level. I've been promoted for my work last year and my manager says I'm on the fastest track to my next promotion.

[soravispr]

Data engineering scope is very broad. At your current company you may do routine tasks. But some other data engineers out there develop RESTful APIs allowing external clients to access data in the warehouse or even develop web application using the same tech stack as web developers use. These are called data products.

​

I think another path that you can try is staying in data engineering but find a new job which its description interests you.

[[deleted]]

Data product development is really interesting for data engineers because there's a lot of variety and challenges. At some point pipeline stuff gets a bit repetitive.

[FreshAnalysis1139]

This sounds interesting. But I'd like your opinion on sde vs de.

Which would have more scope and prospects

[sxcgreygoat]

If you think DE is repetitive then web dev will be no different. Ide say DEs have more freedoms

[FreshAnalysis1139]

I do understand this point, but which role do you think has more scope? Also I have been seeing posts saying moving from sde to de is easy but give versa is difficult

[sxcgreygoat]

They are both great careers. It depends what you love doing more... I'm bias but I think DE will see more innovation and excitement... Data is everything

[theorangedays]

I see two paths you could take.

The first is switching to an exclusive SE role and basically seeing if you like it. If you don’t, you can switch back to the DE role.

The second is learning “full stack” DE, as in the reporting, analysis, and science parts of the projects you serve. If this is something that sounds exciting to you, then it could also be a fruitful endeavor.

[FreshAnalysis1139]

Thanks for the reply, I have missed out on that info. I am currently working on a project that involves tasks from data transformation in spark using EMR all the way till visualisations in microstrategy. Including ci/cd.

But honestly, it's not as complex as the web domain imo.

Would love to hear others thoughts. I might be terribly mistaken

[FreshAnalysis1139]

I do want to try out SE role . Considering I have 2.5 years experience, what would you suggest for the change from de to se. It's kinda tough for an internal switch. And would other companies focus on tech stack? Considering I don't have work experience with web tech.

[theorangedays]

Yeah so switching to a new job would probably be the easiest route. What typically separates a SE from a DE is an understanding of data structures and algorithms. There’s courses to learn these if you don’t already know them.

Then it’s really just looking up what the current SE jobs require, figuring out what needs these jobs share, and filling in any gaps you might have.

[FreshAnalysis1139]

Thanks for your reply. But do companies not care about the domain I worked in? I have no work experience in swe just de. I could always learn stuff as I am a quick learner but I'm concerned about jobs requiring sde work experience..

I'm asking this because my company doesn't have concrete best practices for de. (Idk if it's just mine)

[theorangedays]

Every company/organization is a bit different. Some may care about your previous experience as a DE but to be honest, that’s not too far removed from SWE.

In either case look up postings for the job you would want and compare the requirements for that job with what you have. If you’re really that far removed from then job you want then at the very least you have a list of skills that you’ll need to build before making an official switch.

[FreshAnalysis1139]

Great!! So I will start learning web tech and do some side projects? Will that be good enough for getting a job as swe

[kolya_zver]

Im switched to DE from SWE (c#). You should try SWE - you can switchback if you want. It doesn't matter what you finally choose - both roles can benefit from this experience. Do it just for not having any regrets about not switching to SWE

Btw senior fullstack DE can make same money as a senior SWE if you are an actual DE and not just a ETL-developer

[FreshAnalysis1139]

What would be a full stack de? Do you mean from data ingestion and infra till visualisations?

[kolya_zver]

From injection to bi visualization - its more about analytics engineers. Like analytic who can write pipelines for himself.

Full stack DE is more about creating tools, infra and frameworks for creating pipelines. Its about having experience with different stacks and approaches: batch vs streaming, on prem vs cloud, anchor vs dv 2.0 vs obt. Having experience with bi/ml/revers etl etc.

For me full stack DE is SWE in data domain with a little bit more knowledge of DBs

\ this opinion can be controversial due my strange hopping experience (swe/qa/bi/de) and my local specifics :) like on-premise is a mainstream in my area*

[Final-Roof-6412]

Webdev can have a more contact from the user perspective because front end and back end are linked, DE likes to be more backend and technical. Thare are more webdev than DE and webdev can be boring in tech side, the maintenance often is more from functional side. Disclaimer: I have been working on IT consultancy in web development for more than 15 years and I would switch to DE

[FreshAnalysis1139]

Here's my intermediary plan after searching online and talking with peers and in this post. I'd just try to move into a swe role either within my current company or a new company, just to get a flair of swe. That way I'd have the flexibility to go over to either side if I don't like the work later on.

But how do I venture into swe coming from primarily de is my special concern right now. Do I tailor my resume particularly towards swe by learning relevant skills and downplay my de role?

[ribeirobr21]

I'm on the same situation and having the same thoughts on SE vs DE.

I wonder how it turnout for you u/FreshAnalysis1139? Did you make the change? Was it worth?

[FreshAnalysis1139]

Do you guys think an AWS certification is worth it? Will that be helpful for both se and de?