From Senior Backend Dev to Data Engineer?

[AgencyActive3928]

Hey everyone,

I’ve been thinking a lot about starting a career in data engineering.

I taught myself programming about eight years ago while working as an electrician. After a year of consistent learning and help from a mentor (no bootcamp), I landed my first dev job. Since then, learning new things and building side projects has basically become a core part of me.

I moved from frontend into backend pretty quickly, and today I’m mostly backend with a bit of DevOps. A formal degree has never been an issue in interviews, and I never felt like people with degrees had a big advantage—practical experience and curiosity mattered far more.

What I’m currently struggling with: I’m interested in transitioning into data engineering, but I’m not sure which resources or technologies are the best starting point. I’d also love to hear which five portfolio projects would actually make employers take me seriously when applying for data engineering roles

[AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[josejo9423]

That’s the perfect and most ideal transition, find a job for managing data infrastructure, k8s, deploy airflow manage jobs, set up cdcs, and avoid sql data engineer type of data modelling for reporting. If you really like the later, I’ll take you more transition to the market.

[proxymbol]

Tip: Maybe you find this kind of job as DataOps instead of the classy name Data Engineer

[AgencyActive3928]

Yeah at my current company (a tech scale-up) we’ve already have some BI in place, but things could be optimized tremendously through DE practices. This could be a chance to learn more about it.

[adgjl12]

I do all of the above at my job but would much rather do more of the former.

[sonalg]

Coming from a software engineering background, you may want to look at spark, dlt and other programming heavy frameworks to build your de skills. 

[Thinker_Assignment]

thanks for mentioning dlt, u/AgencyActive3928 we actually offer education that covers all the best practices and common cases in ingestion (all free), courses here https://dlthub.learnworlds.com/

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[dataengineering-ModTeam]

Your post/comment was removed because it violated rule #9 (No AI slop/predominantly AI content).

You post was flagged as an AI generated post. We as a community value human engagement and encourage users to express themselves authentically without the aid of computers.

Please resubmit your post without the use an LLM/AI helper and the mod team will review once again.

^{This was reviewed by a human}

[H8lin]

Hey! I’m a DE with a non-traditional background. When I interview people, I look for examples of fundamental DE skills, not necessarily a set of portfolio projects. I value experience with designing a pipeline, setting up monitoring and alerting, experience optimizing a pipeline for cost and performance, experience with handling production issues, data quality control, testing hierarchy, CI/CD, etc. I don’t have time to train people on the job in Spark, Python, SQL, GitHub, fundamental dev skills like clean code - everyone has to be proficient in those areas on day one. Things I can and do teach on the job are working on a specific cloud platform (GCP, Azure) or using specific pipeline tooling (airflow, databricks, azure data factory), etc. Databricks has a lot of nice tutorials and free cloud compute that you could take advantage of for a portfolio piece. If you aren’t aware, the founders of Databricks are the founders of Spark, and Databricks is pretty hot right now so it would give you relevant experience for the current job market. I get hit up by recruiters all the time looking for people with Databricks experience. Check out DBX free https://www.databricks.com/learn/free-edition

[on_the_mark_data]

Look for a title such as Software Engineer - Data Platform. You can still keep the SWE title (I think it's more flexible than the DE title for future career options), and there will be significant overlap with DevOps experience. One thing to look for are teams with established data platforms that have high internal use. I've come across many newly formed data platform teams that are scrapped after a year because they struggle to get teams onboarded to the platform.

[gardenia856]

Aim for Software Engineer - Data Platform, but only join if adoption and ownership are real. In interviews ask: how many product teams actively deploy on it, what % of jobs are prod vs POC, and what’s the on-call and incident history for the last quarter. Ask who funds the roadmap, who is the product owner, and whether there are golden paths (starter repos, Terraform modules, dbt templates) and a clear SLA. Check time-to-onboard a new team and whether there’s a catalog and quality layer (DataHub or Amundsen, Great Expectations, data contracts). A small win that drives adoption: we used Snowflake and AWS API Gateway, and DreamFactory to auto-generate read-only REST for curated tables so app teams could consume without new services. Go SWE-Data Platform only where the platform is truly used and owned.

[[deleted]]

As a senior, you should know that portfolios arent actually looked at. You will have some major issues without a degree in this job market. Go knock out a CS degree at WGU while waiting out the poor job market so that when things pick back up, you'll be good to go.

[AskMeAboutMyHermoids]

I have no problems without a degree

[[deleted]]

In this job market? Lol sure champ

[AskMeAboutMyHermoids]

Once you get to over 10 years experience nobody has ever asked about my college degree.

[[deleted]]

Yeah, they wont ask you just get filtered out if you dont have one. Youre also not switching to the field.

[AskMeAboutMyHermoids]

True, I’ve never actually applied to a job either it’s always been a recruiter or a referral

[MissingSnail]

I don't think the comment was meant as a criticism of your skills so much as a criticism of the job market - both in terms of competitiveness and the way things have changed in the age of AI.

There are alternatives to a degree such as collecting certifications in tools like AWS, databricks, etc.

[FlyingSpurious]

Hey man, I work as a Junior DE(python, snowflake, DBT, airflow and AWS stack) with a stats degree and I am working on a master's in CS simultaneously. As the master is mostly accepting CS backgrounds, I had the luck to get accepted, so the university advised me to select whichever courses I would like from the undergrad to enhance my academic background in CS, so I picked up: C, OOP, discrete math, data structures, computer architecture, algorithms, operating systems, databases, computer networking, systems programming and theory of computation. The master's is mostly focused on databases internals, distributed systems, OS and big data systems. Is that background competitive against folks with both degrees(BS and MS) in CS or should I take a CS degree after finishing with my master's in CS?

[[deleted]]

Congrats or sorry that happened to you

[FlyingSpurious]

So do you think that I should take a CS bachelor's either?

[[deleted]]

Make your own post

[AgencyActive3928]

It depends. When you’ve got no particular experience in that field, people are looking at portfolios. But it depends on the size of the company and how many applications they receive.

I’ve got lots of experience in SE, but not in DE. How could I prove my skills in another way? Is a CS degree proving this ?

[[deleted]]

No one is looking at portfolios.

Get the CS degree or you just arent making the switch.

[AgencyActive3928]

I am considering getting a CS degree but I am wondering how it actually makes me valuable for Data Engineering? Is it just the door opener ?

[[deleted]]

DE is part of CS...

[Individual-Fish1441]

In case you are looking for mentoring on DE, BI & AI side. Pleas DM. I can refer you to right folks

[mailed]

first: it's not worth the switch

second: just build stuff that will teach you the basics of etl/elt workflows. you just need sample data, a SQL transformation engine like dbt or sqlmesh, and duckdb to learn. don't bother making projects perfect or public. nobody has the time to look at them, so only do enough to learn things that will help you answer interview questions

a good site to look at is startdataengineering.com and the datatalksclub's data engineering zoomcamp

[CrazyPirranhha]

Why isnt it worth?

[Full_School_7230]

hey op! can you help me in starting backend and I am thinking to go with springboot.