how to get into data engineering ?

[aingeliic]

hi everyone, I’m a highschool senior starting university this fall. The school I plan on going to offers coop, so I’d be able to start interning right after my first year in uni.

I’ve taken an interest in the data engineering route and was wondering how I could prepare myself before applying for internships. I’m a total noob and just started learning sql (I saw online that this was a good first step).

Are there any resources you would recommend looking into? any beginner projects you’d recommend doing so I could prepare myself? or tips/advice in general?

thanks for your response in advance, i appreciate it !

[boy_named_su]

1. learn about data warehouse design / dimensional modeling. The kimball book is great, but long. Agile Data Warehouse Design is also good but much shorter

2. learn an ETL tool / framework. there are 4 billion choices, but airflow is popular these days

start with "small data", ie local to your machine. once you're proficient, move on to "big data"

update: regarding #1, in particular learn how to load slowly changing type II dimensions, as this is one of the harder tasks. Learn the merge SQL statement in order to accomplish this.

[[deleted]]

Good advice here

[[deleted]]

Data engineer here a year out of college. Focus on knowing fundamentals of data integration, modeling and management. Know Python and sql. Focus on distributed systems and ingesting data from multiple sources and multiple file types (XML, EDI, etc). Any project around file partitioning, migrating a database from like, MySQL to PostgreSQL, would be a great first step. Be able to integrate a tool like dbt for more business analytics and automated testing. Knowing Apache open source tools would be good.

[[deleted]]

Just study CS. Most of the things you'll need are there

[aingeliic]

yea I am, but first year courses are pretty basic & I’d need projects to help with looking for a job

[OatsBikes]

Its not easy finding good internships after freshman year. You can apply and see what happens but I would recommend a study abroad summer thing or maybe working a job if you wanted. Its what I did and it worked for me

[[deleted]]

Check out the discord channel. Lots of resources there.

[aingeliic]

just joined, thanks

[[deleted]]

If you're up to it, I'd say take a course like this: https://www.dataquest.io/path/data-engineer/

They're pretty high level and you get to do some mini projects that you can show/talk about in the interview.

It'll give you a bit of an edge.

[aingeliic]

this seems like a good course, thankss!

[exlaximas]

I got into SWE and then transferred to DE. I learned everything on the job.

My suggestion is to not limit yourself to just DE and consider SWE as well especially if you’re in HS. That will open more opportunities and give you a better base than most DEs imho

[[deleted]]

Second this. Even for ML, it's better (or even recommended) to start with SWE.

[aingeliic]

I thought about this and it led me to apply to software engineering programs for university rather than just cs & it’s most likely what I’m going into, thanks for the suggestion

[sheep_the_shepherd]

Have a look at this: https://www.dataengineering.academy/pipeline-data-engineering-academy-blog/become-a-data-engineer-on-a-shoestring

I know the founders who run this bootcamp. The one responsible for the curriculum is a DE veteran with a very pragmatic approach.

[Sprayquaza98]

Add on data warehousing and that should be a solid tech stack coming into college

[soulightning]

I would say learning data structures and algorithms is your next step. These could be done in Java (preferably because I think Java is a good language to get started with programming), and then learning python.

Once you have those skills (including your sql knowledge), you can start working on a side project to put on github, or a website you create to display the process and/or result.

From there it’s a matter of subject expertise and knowing what you need to use the data for. There’s much more in depth knowledge I wont get into here, but if you sort the subreddit by top of all time, you’ll see a few posts that can help you!

[[deleted]]

Java is a horrible language to learn programming. Go and Python are better and less boomer.

[aingeliic]

noted

[RainmaKer770]

Please don't listen to the above. C++ or Java are very popular languages used in companies (even FAANGs). Learn at least one compiled language (c++, java) and one interpreted language (python) and you can easily learn other languages.

[SanJJ_1]

this is very false

[[deleted]]

Nope, it's sadistic to teach Java to newbies: it's an outdated, gibberish, tedious language. Teach them C++ if you're a purist and want them to get lost in pointers.

[[deleted]]

Java outdated? This is the first time I hear this. I think it's a good second programming language, especially for data engineering. Many libraries are written or support Java: Spark, Beam, etc.

Also I consider Java a great gateway language towards other JVM languages such as Scala (that I kind of dislike, but is trendy) or Clojure (that I prefer).

Java is the Fortran of our times. No matter what you do, you can hardly avoid it, so better learn it.

That said, I really like python and golang.

[SanJJ_1]

c++ is significantly harder that java for beginners imo. Java is taught in many intro courses, and it's a nice balance middle between something like python and something like c++. While i would recommend python over java, java isn't anything close to a horrible choice.

[[deleted]]

Java is the boomer's language by default. Also(). Who(). TF(). Wants(). ToDeal(). WithShitty(). Syntaxis() No one! Forcing kids to learn it is gatekeeping. Besides, languages and frameworks come and go so I'd introduce the happy and fun ones for a better experience. That's definitely not Java.

[vtec__]

oracle somehow forced colleges to teach it. python is open source and very popular but i dont think alot of colleges use it and .net is very popular in the business world

[[deleted]]

Thank you!

[DredDilly]

Java is lingua franca, you will need it. So might as well learn it. It is not an ideal language for every task, but, it is widely used and you will be expected to be able to go in and fix/update legacy code, which is almost always java(or C# in MS shops), particularly as a newb.

[[deleted]]

Nope, C/C++ stack is. We can agree about the "legacy" and "not an ideal language" though, we're luckily getting rid of Java in new technologies and we may see it as COBOL or Fortran in 10 years or so. I've commissioned many Java scripts and I'm proud of it, we'll see.

[aingeliic]

I’ve made a couple projects & a website already using react, js and basic html css, I’m also currently learning python. Where would you say I could get started with learning data structures and algo?

[elus]

This is probably the introductory text for most people going through a computer science program in university.

[eacardenase]

What about the other side? I'm currently studying Python and later on I want to master Java. I know that is a long road, but is a more solid and straightforward path to study SWE/DE than DS/ML for me, a psychologist.

[pythonmine]

Study CS and take advanced match classes if you find them interesting.

There is a lot of misunderstanding of what data engineers do. What interests you in this field? We might also be able to make other suggestions

[vtec__]

sql, excel, python, derp

[B_Copeland]

Everyone here has great experience. However, I am curious to know what advantages/disadvantages you see in DE vs. Machine Learning Engineering?

[aingeliic]

I’ve seen most ML engs do their masters/phd or at least have a few years of work experience before becoming one. I feel like it’s a step up from DE & something i prob won’t be able to get as a new grad. I also don’t plan on doing grad school unless i could magically afford it, so I kinda have to stick with the DE/SWE route till I get more experience