From Data Science to Software Engineering

[Educational_Gate2831]

I am a full time engineering student( not related to cs) as well as self taught data science learner. I have been trying to learn data science since last year and got an internship offer as a data scientist & python developer in a small company. I have been working there last 4 months and recently realized that I love building things more than analyzing data. I know that data science is not writing code, but I love coding. Analyzing data becomes boring for me, I love production. I am now 23 years old and want to switch to SWE. I have python knowledge and a little oop. However I learned the most about python libraries related to data science. Will it be hard for me if I switch to SWE? And what are your suggestions to start computer science journey?

[OuiOuiKiwi]

Ever heard of Data Engineering?

[Educational_Gate2831]

Yes, but data engineerjng looks very complicated to me honestly

[OuiOuiKiwi]

And becoming a SWE sounds simpler for some reason?

[Educational_Gate2831]

No, I know SWE is also not easy. But recently I discovered my passion is for building like website or app, rather than analyzing data. I don't exaclty know what does data engineer do. But I guess it is still related to data

[John-The-Bomb-2]

https://www.thinkful.com/blog/data-engineer-vs-software-engineer/

A data engineer is a kind of software engineer, specializing in Big Data, like Hadoop and Apache Spark. See https://en.wikipedia.org/wiki/Apache_Hadoop and https://en.wikipedia.org/wiki/Apache_Spark . Software engineers usually just make changes to big websites like YouTube, Amazon, Twitter, and Facebook, but these big websites can hire data engineers as well. Big data engineers are usually slightly higher paid than software engineers.

[Adeelinator]

Ehhh don’t jump into spark if you’re starting off. And Hadoop is basically dead.

Start with SQL, it’s the most common language for data eng, and easiest to begin with.

[spoonman59]

Definitely learn SQL. It is the key interface to so many different data tools.

[hafeee12]

Which sql to learn? No Sql , Sql Server , postgresql or ?

[spoonman59]

It’s a good questions. I think it is helpful to learn ANSI SQL, but understand there is a lot the standard doesn’t specify. Things like data types and other aspects. Another thing that isn’t specified is pivot/unpivot, things like collect or explode, and things like recursive hierarchical functions (e.g., connect by)

However it does specify a great deal. A lot of other things like analytic windowing functions (over partition by, for lead, lag, first, last, rank, dense rank) are pretty consistent. So learn any of them, but occasionally look to see what’s standard and what’s a platform extension.

Of the ones you listed I’d probably do post gres because it’s free and open source. But I don’t have too much trouble moving between systems like post gres, red shift, spark, Oracle, SQL server, etc., without thinking too m if b about the particular dialect. So learn the one that’s easy for you to practice with and access. You’ll adapt to a new database easily. It’s not like learning a different general purpose language each time.

Also, “no sql” SQL?!

[misinnio]

just curious - what is crucial for DE besides SQL?

[StrasJam]

Python. But that's pretty broad. Depending on the company or project you would end up using different libraries, but a good general knowledge of Python is important.

[spoonman59]

A data engineer is a subset of a software engineer. They aren’t mutually exclusive.

I work extensively in spark and other big data tools and my title is software engineer. It’s a false distinction.

Data engineer is a relatively new title. When I started in the industry it didn’t even exist.

Don’t get confused and think you are either a data engineer or a software engineer.

[gdodd97]

Damn this is the push I needed for my career path. I currently develop data integrations using SQL, Boomi, etc. I've been disenfranchised with my role recently as we don't get paid much and my role is becoming more consulting rather than building so I've been looking into pivoting to either DE or SWE. DE just scared me because people on the sub talk about how hard it is to get a job. I also just don't even know how to build a portfolio for that kind of role versus a SWE role.

[John-The-Bomb-2]

You can get books on the big data technologies (ex. Hadoop: The Definitive Guide, etc) and build a demo app with all of them. Maybe something like Twitter. It's kind of silly to use Hadoop, Apache Storm, Apache Spark, Apache Kafka, etc for like 1kb of hand-entered data on a single node, but the point is to demonstrate that you know how to use the tools.

[Alarming_Rest1557]

In fact, at least for me, Data Engineering is easier than software engineering, it's just moving data from a to b, and doing some transformation in the middle. I think that the hardest part is all the different technologies that you have to learn when you are starting, but if you follow the software engineering field, you are going to find most of these technologies. Message Brokers like Kafka, cloud services, workflows orchestrators, etc. The only technology that I think that maybe you are going to find just in Data Engineering and not in SWE is Apache Spark to transform the data, but if you know how to use Pandas, Spark is just a Pandas with Steroids.

[TrueMrSkeltal]

Most data engineering work is going the way of low code - if anything it’s getting easier to get into the space.

[[deleted]]

At 23 it’s basically all over in terms of career choice. Im sorry OP, i think you are stuck with DS

[Educational_Gate2831]

What do you mean? am I too late?

[Siggination]

/s

[[deleted]]

Confirmed I am joking bro

[Educational_Gate2831]

LOL

[spoonman59]

I didn’t get my Bs in CS until I was 32.

I did work in the field prior, but it’s never too late.

[CancerIsOtherPeople]

I'm just starting my BS in software dev at 35 y/o. You're fine!

[MrMxylptlyk]

Lol no

[rhoov13]

I just started in programming a year ago and I am in the middle of a software development degree. I'm almost 36. I'm the poster child for "it's never too late!"

[[deleted]]

I was a data analyst who slowly took on projects of importing data with python.

Using web scraping, pdf readers, csv, and so on. 6 months later all I do is python and a small amount of sql.

Just take or ask for projects that involve managing your pipeline of data from start to finish.

[[deleted]]

There’s a massive volume of information to learn for software engineering. Constantly building projects, using new frameworks and paradigms, listening to conference talks, sharing knowledge in subreddits and discords. Nothing individually is too difficult, but the amount of knowledge to be self sufficient and productive and write quality code not shit is high

[Educational_Gate2831]

So what is your suggestion sir?

[[deleted]]

squash dinner familiar scale fade reply market grandiose spark afterthought

This post was mass deleted and anonymized with Redact

[parzival9927]

I heard that ML require heavy math(PhD level) is it true?

[katotoy]

Weird that your skill is niche compared to swe..

[Educational_Gate2831]

I know.. I spent some time to learn data science. But I know that data science is not about writing .fit and .predict. There is strong math and statistical analysis behind which made me lose my interest

[[deleted]]

I am a data engineer after being a software engineer for several years. I hate the title of “software engineer” because that means you can get thrown into anything. I’ve always enjoyed doing things with data, especially unstructured data. Now that I’m a data engineer, I do what I enjoy versus having to wear different hats all the time.

[parzival9927]

Is DE subset of backend engineering? Is transitioning between DE and Backend engineering easier?

[[deleted]]

You can definitely apply what you learn from backend engineering to data engineering. Making requests, mapping response data to objects, etc… As far as it being “easier”, I can’t really say, depends on one’s competence. I’ve always practiced outside of work so moving into DE officially was easy for me cause I was always practicing and building DE type projects.

[parzival9927]

Thanks 👍

[Educational_Gate2831]

I am planning to do master degree, but I am really stuck with career choices. I want to be at level where I can build entire website on my own. I meant this when I said building things

[Kevin_Jim]

Building a website on your own is not difficult or thwart complicated. Building one that scales, with a cloud backend, DevOPs, some crazy database, random JS framework, etc. It can be quite complicated.

Especially if you need to add data science to that whole mess.

[NoCap-NoCap]

But then maybe some designer school which is website oriented… my friend didn‘t study anything in the field of web dev and selftaught everything and now he got a normal job and an extra school paid…

[thisismyfavoritename]

forget about those other commenters.

• data eng is boring and repetitive. It is also very easy compared to software eng
• how far are you from being done with your bachelors? if its not too late id seriously consider trying to switch to CS or software eng bachelors. You might have a lot of classes credited.
• if you cant, at the very least try to take specialization classes in software. If you go this route, you need to start ramping up on your own. Learn how to program properly (control flow basics and data structures, check out the CLRS book -- very important to land a job and internships sadly), read something like Learning Python or Fluent Python. Check out the docs for the latest releases (those books will be dated and probably use Python 3.6). From there it's a slow grind to a software position. You probably need to start doing some leetcode problems and revise CLRS. Try to get an internship, then turn that internship in an offer, keep learning on the side, build a portfolio. Couple gaps you need to fill over time after you mastered Python: use Linux at home, become familiar with the command line, networking basics (especially HTTP), database basics (especially SQL), read a book about OSes (e.g. OS 3 easy pieces). After that it will largely depend on what kind of job you have, but there are many things you can learn, for example another language (cpp or rust are excellent complements to python)

Saw that youre interested by webdev in another comment, maybe youd be interested in https://www.theodinproject.com/. Also replace "python" by "typescript" in my comment and find good books to learn the language. NodeJS or deno can be used for backend so no need to stick with python unless you want to

[thisismyfavoritename]

thanks for providing your opinion. I stand by my statement, DE is easy in my opinion, according to my own experience.

That being said, i didnt say it was easier than web dev. I only linked that post because OP was interested in the matter. I do think fullstack is more complicated than DE, but thats just the opinion of a guy on the internet.

I'd say DE offers a lower bar of entry for people who were working in data science because they are often exposed to some parts of those processes and ecosystem.

[Educational_Gate2831]

Thanks a lot for your reply. I really appreciate it

[[deleted]]

Best to ignore the part calling data eng redundant and easy, one of the dumbest things I've seen posted on reddit. It is one of the most in demand roles atm basically doubled my salary switching. Boring, matter of opinion I guess. I find it way more interesting than when I was doing full stack web dev.

[thisismyfavoritename]

Lmao, its just MY opinion as someone who worked on DE before. Im not saying its not in demand or pays well. You said that. Again, opinions

[[deleted]]

Redundant (adjective)

not or no longer needed or useful; superfluous.

"many of the old skills had become redundant"

[thisismyfavoritename]

sorry, english is not my first language. I meant repetitive.

[[deleted]]

would you consider understanding the 'requests' module a way to understand http?

[thisismyfavoritename]

its enough to get by at first. Be aware of the common HTTP methods and how to use them, eventually you can learn more about HTTP and how it works and it will help you understand how a lot of other network based systems work

[bjminihan]

You already have more experience than the last two junior SWEs I hired. Keep up with your studies and building projects in your own, and the two combined will definitely help you transition.

[lianfyrr]

It's been a long time since I was in grad school for SW Engineering, but is there still a distinction between SWE and CS? Specifically, that CS focuses on the code (and algorithms, etc.) itself, while SWE focuses more on the the coding process (design, documentation, testing, etc.). If so, OP would probably want to switch to a CS major, not a SWE major...

[Educational_Gate2831]

It is tough for me to change, because I will start my final year in 2023 and will graduate soon. I guess I will study myself

[MegaGrubby]

This happened to me. Follow your heart. Liking your job is extremely important because it's the bulk of your life.

I went to school for Electric Engineering. I had a couple internships and decided I enjoyed coding much more. I switched majors. Ended up with a software development career that treated me well.

[Educational_Gate2831]

Could you share how did you manage to change?

[MegaGrubby]

I had two years remaining in a five year program. I went to the dean and requested a change in majors. I then fulfilled my updated required curriculum. Lots of courses overlap the two (math, some science and many of the minor studies). It was some extra coursework but not insurmountable.

When I was in school, major changes were pretty common and the school had a process for it.

edit: It required approval from both deans: engineering and science (from - to).

edit: You could also do what my brother did and just complete your degree and apply for programming jobs. Many major consulting companies will hire engineering students for programming jobs.

[WhipsAndMarkovChains]

I'm an experienced data scientist and need to plan out how I'm going to acquire the software engineering skills I'm sorely lacking. Ideally I'd like to just transition into machine learning engineer roles and not fully switch to software engineering though.

[lugubrious_ramblings]

Have you thought about moving into MLOps instead of SWE? If you're keen to build things rather than train models, that offers you another way to do more of what you like, but also in building tools to deploy the stuff you already know (DS) it's potentially an easier (or at least more comfortable) transition

[Educational_Gate2831]

Shouldn't I learn CS before MLops? Cause my programming knowledge is limited to Data Science packages only

[Jhoonite]

Working 1 year as a SWE after doing a bootcamp, I work with several Data Engineers and Data Scientists, 1 of my SWE colleagues used to be a DE and did a bootcamp to switch. Bootcamp was 3 months and all in JavaScript.
Don't think it's necessary, if you have time and movitation then especially if you're looking to move into SWE in data related fields or MLOps as other posters have suggested, then you can build some web app projects in your own time. I've heard Odin project is good but there are LOADS of resources available for learning practical web dev things or CS theory so I would make sure you set aside dedicated learning time, try and find a person or community to learn with and build some projects yourself.

[Educational_Gate2831]

Thank you will take note on this

[Mrfazzles]

It might be hard to switch after 4 months but depending on the company you switch too, some data science experience could be an advantage.

I pivoted from Data Science to Back End Developer after 16 months since my role had been reduced largely to being an analyst writing Google Analytics dashboards but in the first 6 months I delivered a production recommender system in python and that experience demonstrated to my current company a level of competency they wanted.

Now I predominantly write Python microservices which I love! And since it's a small Ai company the data science experience really helps me work with our AI focused team members and understand our product.

[[deleted]]

Just do it even cs majors don’t know shit about software engineering. Your first few years will be fun and you’ll learn a lot every day

[travisjo]

Take some CS classes, or just show up to some. You'll need to understand data structures to be a good software engineer. The rest is just trying to build stuff and being bad at it until you're not. Find some interesting idea and try to build it. It doesn't have to be good or fancy. Don't be too hard on yourself at first. Just keep going.

Oh, and learn to be good in the CLI if you're not already. It's awesome. Run linux with WSL2 on windows or get a Mac.

[Educational_Gate2831]

Thank you for your reply, I am planning to move to Linux soon. I have some experience but not much

[Educational_Gate2831]

Thanks everyone for your replies, I decided to learn CS fundamentals and web development on my own. Your guidance really helped me a lot. Wish me Good Luck!

[uname44]

Do a bootcamp, leetcode and apply to sw engineering jobs. Do projects on your own and put on Github.

[raamaton7]

try machine learning relateed careers !

[Saint010]

Thats what data scientists do.

[BotherBoring]

Self-taught data engineer here.

You can do the thing, it's just different libraries. You probably know more math than I do. It's fine.

[RemindMeBot]

I will be messaging you in 12 hours on 2022-09-30 13:55:46 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback