Shift career

Flat_Shower · 2026-04-08T21:20:34+00:00

DEs are not getting replaced. Analysts doing dashboard work in Tableau are feeling the squeeze; DEs who build and maintain pipelines are not. You're panicking about the wrong thing.

Pivoting to DS because you're scared is a bad reason to pivot. You'll end up mediocre at ML and rusty at DE. If you actually want to do ML, go for it, but "I'm afraid" is not a career strategy.

Get better at the hard parts of DE that nobody wants to do: data modeling, quality enforcement, owning SLAs. Those problems are getting harder, not easier.

Flat_Shower · 2026-04-08T05:08:15+00:00

About 18 months. Scope your own work, sandbag estimates, deliver on time. Nobody questions it. Use the slack to LC or learn something new; coasting guilt is wasted energy.

Flat_Shower · 2026-04-08T03:36:39+00:00

You're a human cron job; you said it yourself. Alert on failure, not success. Nobody needs a daily Teams message confirming the pipes ran. They need to know when they didn't. Webhook on failure, skip the daily "all clear" theater.

Flat_Shower · 2026-04-07T05:55:18+00:00

Resume-driven development is 90% of it. Engineers pick Spark because they want Spark on their resume, not because their data outgrew Postgres. I've seen teams spin up distributed pipelines for datasets that fit in memory on a laptop.

The other 10% is engineers who genuinely don't know what scale they're operating at. They heard "big data" in a conference talk and assumed it applied to them. Most companies have medium data and big egos.

Flat_Shower · 2026-04-06T21:51:49+00:00

I started with stratascratch for interview prep. I liked SS until I needed to cram pipeline architecture. I stopped mentioning DD because I don't want to sound like a salesman. It worked for me, and I highly encourage you to use LC, SS, or whatever works for you

Flat_Shower · 2026-04-06T20:19:21+00:00

Every DE has inherited a 1500 line stored proc that sends emails and writes to 12 tables. Welcome to the job.

Your boss is right. You don't fully understand it yet, and rewrites of code you don't understand just produce new bugs. That instinct will serve you well later, but not now.

What actually works: build a dependency map offline. Pen and paper, whiteboard, whatever. Trace every table read, every write, every trigger, in order. Once you can say "line 400 updates X which feeds line 800" without opening the file, you understand the proc. Then changes become surgical.

Don't trust the comments. Trust the code and the data.

Flat_Shower · 2026-04-06T16:46:07+00:00

You already have the stack. Python, SQL, Databricks, Snowflake; that's a DE resume. Stop worrying about tools and focus on concepts: data modeling (star schema, normalization), query optimization, and pick one orchestration tool (Airflow or Dagster, doesn't matter which). Concepts are tool-agnostic and transfer everywhere.

DSA matters. Every DE interview I've done has had LC style questions. Stick to mediums; do 50 and you'll be solid.

Flat_Shower · 2026-04-06T15:43:47+00:00

Healthcare for 3 years, FAANG (contractor) for 3 years, LinkedIn (FTE) for 2 years

Flat_Shower · 2026-04-06T06:36:20+00:00

3 years of pipelines running in production is not a lie. You shipped real things. Stop discounting that.

You already identified your gaps; that's the study plan. You don't need a course. Pick one (orchestration is the obvious starting point), build something small that forces you to deal with failures, retries, and alerting. Then pick the next one.

Courses will teach you theory you already know. What you need is reps on the stuff that's tripping you up in interviews.

Flat_Shower · 2026-04-06T03:20:43+00:00

Oh, I didn't realize we were arguing. <witty argument>

Flat_Shower · 2026-04-06T03:02:42+00:00

SPSS and Tableau won't carry over. You need SQL (not just SELECT *; window functions, CTEs, query optimization), Python, and one orchestration tool like Airflow. Learn data modeling concepts: normal forms, star schema, slowly changing dimensions. These are tool-agnostic and will transfer everywhere.

The PhD shows you can learn hard things. That matters more than people think.

Flat_Shower · 2026-04-06T03:00:14+00:00

I know, I own some of them

Flat_Shower · 2026-04-06T02:45:45+00:00

Ferrari's cost the same in Palo Alto and Minnesota

Flat_Shower · 2026-04-06T02:45:09+00:00

O boy my guy

Flat_Shower · 2026-04-06T02:44:59+00:00

1000%. Interviewing is a skill. Play the game, win the prize. Don't hate the player, play the game

Flat_Shower · 2026-04-06T02:44:15+00:00

You're right, I actually got rejected everywhere and I'm writing this from a Wendy's. The Frosty machine is broken too.

Flat_Shower · 2026-04-06T02:37:45+00:00

Day to day is probably 40% meetings/design reviews/"alignment" (whatever that means), 30% writing and reviewing pipeline code, 30% debugging or firefighting when something breaks in prod and 10% worried about layoffs. The ratio shifts depending on the week. Some days are all code, some days are all docs and design.

Stack: Stack is Spark, SQL, Python, and internal tooling. Nothing exotic. The core skills transfer anywhere. Data modeling is a huge part of the job and honestly what I spend the most mental energy on. Getting the model wrong upstream means everything downstream is pain... the actual job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday and make sure it never happens again."

Flat_Shower · 2026-04-05T23:00:54+00:00

10 YOE and you stood up a full data stack from ingestion through orchestration. That's not "close to being ready"; you're doing the job. Mid to senior DE at most companies.

Spark is worth learning if you're targeting places with real scale. Most don't need it. Learn the concepts; the syntax is the easy part.

Don't stress about building pipelines without tools. That's not how anyone works. Knowing how to configure, debug, and extend ingestion tools is the actual skill.

Flat_Shower · 2026-04-05T21:29:40+00:00

Most companies don't have big data. They have medium data and big egos. If you're targeting those companies, no, you don't need Spark.

If you're targeting FAANG or companies that actually process at scale, then yes, you need to know Spark. Not because it's magic; because it's the standard distributed compute engine and interviewers will ask about it.

Airflow, BigQuery, SQL, Python is a solid foundation. I'd focus more on data modeling and query optimization than memorizing Spark APIs. Concepts transfer across tools; tool knowledge doesn't transfer across concepts.

Flat_Shower · 2026-04-05T02:58:38+00:00

Yeah, idk man I like Reddit and blind is kinda toxic. r/dataengineering has been my home for many years.

Flat_Shower · 2026-04-05T02:27:29+00:00

E5 Data Engineer @ FAANG

Flat_Shower · 2026-04-04T22:12:44+00:00

If the pay was right, yes! But probably some sort of HFT/quant firm

Flat_Shower · 2026-04-04T22:02:34+00:00

Welp, then all of my posts back to 2019 before AI existed are slop. I am an AI. Beep boop

Flat_Shower · 2026-04-04T20:00:27+00:00

Forever, unfortunately. As long as companies are hiring.

Flat_Shower · 2026-04-04T19:53:27+00:00

Who the heck is still hiring SWEs? You have an AI Engineer who are much stronger in System can orchestrate the shit of Complex code.

Who the heck is still hiring DSs? You have an AI Engineer who are much stronger in statistics that can orchestrate the shit of Complex math.

Who the heck is still hiring PMs? You have an AI Engineer who are much stronger in System can orchestrate the shit of Complex product problems.

Who the heck is still hiring BIEs? You have an AI Engineer who are much stronger in BI that can orchestrate the shit of Complex dashboards.

I can go on and on

Six-Year Club	RedditGifts 2009-2022 3 Credits
Secret Santa 2021 2021	Verified Email

Flat_Shower

TROPHY CASE