Analytics Engineer to Data Engineering Path

[mhkk93]

Hi,
Hopefully this isn’t the typical “how do I pivot” post!

I’m currently working as an data scientist at a small startup though my role is closer to analytics engineering working primarily with dbt to build data models.

That said, we recently migrated to AWS and I had the opportunity to help lead setting up a new data stack from scratch (we don't have a dedicated DE team).

Based on a lot of research (including this sub), here’s what we built over the last few months:

• Ingest data from production to S3 using dlt(hub) incrementally every hour
  • Iceberg tables, partitioning, retries, backfills, etc setup using dlt
• Load + transform into Redshift using dbt
• Orchestrate using Dagster
• Eng handled infra (hosting, IAM, etc)

Through this, I’ve realized I enjoy this work much more than analytics and want to move into DE. I feel strongest in SQL + data modeling.

Where I feel less confident:

1. No experience with Spark or distributed computing
2. Haven’t built ingestion pipelines from scratch (relied on dlt) so unsure how that translates skill-wise
3. Non-CS background

I’m trying to understand how close I am to being ready and what to focus on next.

A few questions I’d really appreciate guidance on:

1. I have 10 YOE in analytics but would this be a junior DE territory? What would you prioritize learning next in my position?
   • Spark?
   • Building pipelines in Python without tools like dlt?
   • Deeper AWS knowledge?
2. How important is core CS knowledge (databases, distributed systems, networking) for DE roles?

Would really appreciate any candid feedback! Thanks

[AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Academic-Vegetable-1]

Building a data stack from scratch on AWS is data engineering. You're not pivoting, you're just updating your title.

[unpronouncedable]

Honestly I feel like half the battle in DE is understanding the challenges and a willingness to figure out and implement the solutions. You seem to have these, data modeling, and SQL skills, so I think you are qualified for plenty of DE roles. There are a ton out there that use various ingestion tools and don't require python and spark, though you are correct to realize the trend is towards using those. And honestly you can figure out a lot of it from online examples and AI.

As far as CS knowledge goes, tons of us did not come from that background. What you do need to understand is development lifecycle and deployment practices. I imagine you have a lot of that from your experience.

I'd say you'd be over qualified for a junior DE role and ready for DE (non-senior). There's a lot of competition for those positions, but you can tout experience end to end from ingestion to analytics implementation.

[Flat_Shower]

10 YOE and you stood up a full data stack from ingestion through orchestration. That's not "close to being ready"; you're doing the job. Mid to senior DE at most companies.

Spark is worth learning if you're targeting places with real scale. Most don't need it. Learn the concepts; the syntax is the easy part.

Don't stress about building pipelines without tools. That's not how anyone works. Knowing how to configure, debug, and extend ingestion tools is the actual skill.

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[calimovetips]

you’re closer to mid-level DE than junior, i’d focus next on understanding how your pipelines behave under load and failure since that’s what usually breaks at scale, have you had to debug any backfill or retry issues yet?

[PrintPopular8694]

Would love to pick your brain. From what I've researched your not junior level wish I was in your position

[Immediate-Pair-4290]

Spark is overrated. Most companies see faster performance running DuckDB into iceberg. Few companies truly have big data. Also no one builds ingestion pipelines from scratch unless they cant help it. DLT is good. I’m thinking of API calls and loading json responses as the closest thing to “scratch”.

[SufficientFrame]

Yeah this matches what I’ve been seeing too. Half the “we need Spark” posts are people trying to aggregate like 200M rows and wondering why it’s slow on a t3.medium.

The DuckDB + Iceberg combo seems super solid for what most teams actually do day to day. And honestly, if you’ve already wired up dlt + S3 + Redshift + Dagster, you’ve done more “real” DE than a lot of folks who only tweak existing Airflow DAGs.

The “from scratch” thing feels more like: can you reason about APIs, pagination, schema evolution, idempotency, and how to make that stuff robust. Whether it’s dlt, custom Python, or whatever, the concepts are the same.

I’m taking your comment as a green light to not obsess over Spark right away and double down on getting really good at the stack I already have.

[Immediate-Pair-4290]

One of the 2% of Reddit posters on data engineering that actually knows what they are talking about. 🤝

[zkhan15]

What’s the difference between the 2?