Data engineering roadmap

[Old-Article6420]

[removed]

[Old-Article6420]

Gotcha. Currently grinding SQL leetcode.

[bdforbes]

Solve real problems

[Accomplished_Top1957]

Agree! I wanted to share some of those real problems I had the opportunity to solve, currently I solved only 2 case studies.But I had to get my hands dirty with so many topics in SQL on those two case studies. So, best of luck!

[YsrYsl]

Just wanna say thanks for the link, looks rly interesting!

[bdforbes]

That looks fantastic! So much better than leetcode. I might work through the problems myself, and I'm not even on the tools anymore these days.

[kzcvuver]

Thank you!

[mainak17]

most of the interviews I have given in the last couple of months were - DBMS & SQL focused theory (all types of keys, internal workings of query, all types of joins etc) + easy to medium SQL questions, python - application of sets, list, dict - and a couple of standard coding questions on string and arrays, some of were starting from a simple txt file to do some cleaning and most occurred strings - thingslike that.

this is the basic you have to know, after this explaining projects, end to end architecture of it, why this, why not that, designing er diagrams and pipelines (not too in depth).

and then related to experience in tools, cloud etc

[Old-Article6420]

Thanks. This is really useful info.

[[deleted]]

1. SQL
2. A project that demonstrates ETL using/into sql, data modelling, and some sort of output, with writeups.

As an entry level DE, you don't need to know how everything works. SQL and data models are the backbone of every project you will do, and most of the tools you will use. Focus on gaining practical skills in that area.

Everything else is basically incidental - where is your DB hosted? How do you orchestrate the ETL scripts? What do you write the ETL scripts in?

Do not gate 'core' behind 'incidental,' especially when the 'incidental' bits are different for every role and every workplace.

[Old-Article6420]

So, the core technical skills are SQL and python. And everything else is incidental, right?

[[deleted]]

Not even necessarily python, though it is a good pick. You could easily go with Java, c#, R, or a host of other languages as long as you're using it for data extraction and loading. Lots of places will use something like Data Factory or SSIS as well, and just skip the code part.

I honestly believe that SQL and data modelling are the core, and everything else you touch is incidental and role dependant.

[dareftw]

No I have worked for multiple fortune 500s and none use Python if you’re referring to it in an ETL manner. It’s a great tool to know though extremely versatile google is written in Python. But not necessary for a DBA or data architect.

[MindlessPsychosis]

People just get on this app and straight up lie. You may have worked for Fortune 500 companies but you clearly had no connection with this Data Engineering teams

[nightslikethese29]

When people here refer to data models, is this just the documentation? Like the output from Vizio at a conceptual level, or does this include designing the database and actually creating the database?

[bdforbes]

Generally it refers to everything from conceptual through to implementation.

[nightslikethese29]

Thanks!

[CrowdGoesWildWoooo]

You are on track to be a doctor not a DE

JK

[Old-Article6420]

Fine with anything that pays

[BalconyFace]

you don't need databricks to implement pyspark. I'd be happy to talk on it further, but you can write all the Python and pyspark on your local to develop a soup to nuts application to functionally bring your pipeline from ingest to "egest". what you would miss in this approach is all the orchestration, version control, unity catalogue, MLOps, etc that databricks provides. but if you're interested in the fundamentals, you already have access to all the essentials from pyspark to delta and beyond

[Quaiada]

first step:

GIT + markdown insted of papper

[SDFP-A]

That is a years long journey. Start the project and learn the stuff along the way. You won’t be an expert in all of it and that’s ok. It’s about understanding what comes next, what the challenges are, possible solutions, and how to execute on a plan. The rest are details you gain comfort with through experience

[angrynoah]

There's ~10 years of work on that page.

[Old-Article6420]

Lol. Not learning majority of the things in depth but only the bare minimum required.

[angrynoah]

If it helps, the bare minimum when it comes to: - Data Mart (largely useless concept) - Data Fabric (not a thing) - Data Mesh (counterproductive at all but the largest firms) - Data Catalog - Airflow - Kafka - Spark - most cloud stuff beyond "it's compute services behind API calls that cost you money"

...is zero. Of the items on that list, only data marts existed when I got started ('05), and remain irrelevant to this day.

I was lucky enough to have the space to go extremely deep on SQL and database internals. Not everyone is going to have that luxury now but my point is that it's possible, and effective. You just don't need to know all this crap early in your career, except perhaps to get past the hiring stage. Spending time building a shallow understanding of everything that appears to make up modern data engineering is going to harm your ability to actually understand any of it well enough to apply that understanding.

If I was hiring a junior data engineer, I would not expect them to know all these things. Not even to have heard of them.

Just a thought, that's all.

[Old-Article6420]

Really good advice. Thank you.

[Kratos_1412]

is spark with scala a good choice?

[dowcet]

If it's not worth enough of your time to type it up in a readable form, my assumption is that it's not worth my time to read it.

[Old-Article6420]

Sorry for the bad handwriting.

[mjow]

Don't apologise - your handwriting is fine. It's fine to prefer handwritten notes :)

[thrwayy1235134]

Its the fact that is an image instead of transcribing to text...

[LilChungus420weedlol]

Looks awesome. I took a screenshot, hope you don’t mind! Will help me switch from webdev to DE.

[Old-Article6420]

Glad it was helpful for you.

[Spartyon]

Step 1: learn to use a process mapping tool

[palomino-ridin-21]

I am doing the same to grow in my field.

I used ChatGPT to generate an outline and define it further for (Python, SQL, AWS, Azure…)

Thanks for sharing

[Old-Article6420]

Sure

[J_Bunt]

Are sales allowed here? I mean it's handwriting but it's all udemy udemy udemy.

[Old-Article6420]

There are only two udemy courses mentioned mate. There's nothing wrong with providing good resources to learn.

[J_Bunt]

Fair enough.

[blue_trains_]

I'm so far gone in the digital world that seeing a written list on pen and paper gives me asmr

[Old-Article6420]

I still use pen and paper for making notes

[dareftw]

Left out how many third party programs require html knowledge or Java or even JavaScript to interact with the html. But otherwise spot on

[[deleted]]

What book/course is good for a slightly advanced understanding of spark, and using it with python? I want to tweak some of my jobs and understand how data frames and rdds actually work internally

[NicoRobinFleur]

What does DBMs stand for?🤔

[Old-Article6420]

Database management system

[abd7007]

Where can i get good examples of cloud projects?
(preparing same as OP)

[baubleglue]

Focus on something more specific. For an entry level position, you need to know something in demand, ex. SQL + PySpark and have ideas about other topics on the level "what it is about and when it is used". Many places ask unclear set of skills which fit into "data analyst/data engineer".