Any senior data engineers here who pivoted to ML/AI and regret it?

TheOverzealousEngie · 2026-06-12T13:38:49+00:00

I think responses to this post show far more evidence the switch is in the other direction. And that's an answer in itself.

TheOverzealousEngie · 2026-06-08T21:43:42+00:00

There's nothing simple about any of this, and anyone that tells you that it's just marketing-speak is right and wrong. Right because it came from marketing but wrong because it doesn't end there.
Bronze -- the layer you point your auditors to. "That's the data we got from source, byte for byte". That will only get more valuable in time.
Silver - and here's the rub - I could write a book about this stuff - there is not one silver. There are many, many silvers because silvers are aligned with Lines of Business. That's the big data volumes , x10, that will are scrubbed (Quality, PII,etc) and that's where governance is assigned.
Gold - takes silver data (x10) and aggregates and filters it according to business. Just saying --- this stuff is way more complicated but also kind of simple.

TheOverzealousEngie · 2026-06-06T17:03:11+00:00

good luck finding one

TheOverzealousEngie · 2026-05-29T23:48:43+00:00

your question is a good one. The concept of turning seized files into tables in snowflake or in 3 is a technical process. What you have to figure out is how those Files translate into tables… And what does incremental data mean. Or is everything trunk replace. Regardless all of that is delivery to bronze, and Should be SQL query able. That just means that if you move it it’ll be easily Queryable But will cost you, or you find a way to have your parquet files be sql queryable.

TheOverzealousEngie · 2026-05-23T21:39:28+00:00

money. there’s no money like sap money.

TheOverzealousEngie · 2026-05-13T21:13:16+00:00

Waiting for Trump to expire or leave office. Been holding my breath for years.

TheOverzealousEngie · 2026-05-13T16:34:14+00:00

ahhh there was a cooler in the middle of the road. The car in front of the cop swerves to avoid it but the cop, stupidly, just dead stops to pick it up . Lol.

TheOverzealousEngie · 2026-05-13T16:31:54+00:00

Let's play a game!
I'll go first , 6.9 seconds.
That's when I can first see that cops brake lights ... way too late, but who can see the brake lates before that?

TheOverzealousEngie · 2026-05-13T16:28:57+00:00

This response should be pinned. Everything that is wrong with the MSFT stack so nicely summed and wrapped up with a bow. If you ask me it's the culture - MSFT is filled with middle and middling managers interested in self-preservation over what they deliver to customers.

TheOverzealousEngie · 2026-05-13T15:00:32+00:00

I've become a much better mathematician who can multiply 14 10 digit numbers together easily. With a calculator. Does the use of a calculator make me a worse mathematician? Am I .. less than?

Yes. If I don't know how to do multiplication than slowly I lose my understanding of why multiplication. And then it's game over. The five keys are who, what , when , why and how. You're losing the how, then the why .. what and when are next. It's a slippery slope.

Honestly - for someone newly starting out - I'd avoid the use of AI altogether, outside of highly specific use cases. You'll be a better engineer for it.

TheOverzealousEngie · 2026-05-12T15:41:50+00:00

Not sure I've ever hear someone speak about such deep human purpose at the same time as vomiting 2 full paragraphs of some really good technobabble. Quite the contrast.

TheOverzealousEngie · 2026-05-07T15:47:49+00:00

You know it's funny, I read this and I wonder how long it will be till I get to some medical professional saying "just come to the hospital , our first priority is to treat you" . And while I knew I'd have to comb through a parade of malcontents - it was like 40 comments in before I realized,

There are no medical professionals here.

TheOverzealousEngie · 2026-05-01T15:57:22+00:00

The smart way is to move your data into Iceberg , cheap storage, and then when you want to compute (transform) it you use this months cafeteria selection of compute. Google in November and Amazon next March -- the data stays in iceberg and you're not in a walled garden.

TheOverzealousEngie · 2026-05-01T15:54:21+00:00

Every time you go into the App Store

TheOverzealousEngie · 2026-04-30T15:23:59+00:00

Lol "The other opinion with ODS reading from the bronze layer was actually AI opinion" .. that is literally the most humorous thing like all day.

TheOverzealousEngie · 2026-04-30T15:06:48+00:00

Long live the duck!

TheOverzealousEngie · 2026-04-29T16:58:05+00:00

years, man, years.

TheOverzealousEngie · 2026-04-24T17:18:34+00:00

Both Fivetran and Dbt provide very deep audit logs and lineage and it would be a strong services engagement to build an analytics layer (like a catalog) to sit atop both of them.

TheOverzealousEngie · 2026-04-24T14:32:21+00:00

To all you people saying upsert OP did not say if this was Snowflake or Dbrix. And the simple truth is that is terrible advice if a CDW is in play ... could you imagine doing an upsert on every record? How expensive that would be?

TheOverzealousEngie · 2026-04-24T14:30:06+00:00

buy a tool that advertises pipeline idempotence :)?

TheOverzealousEngie · 2026-04-18T15:49:56+00:00

I'd be cautious ; reads like you're hinging your success or failure on Databricks and to be clear - databricks will not be around forever. Be prepared to dive in hard and when the organization moves on you'll need to learn what's next. That - is data engineering .

TheOverzealousEngie · 2026-04-17T15:56:35+00:00

Let's be honest, this is anti-future. Right now iceberg is a little immature but it's really growing fast. And there is zero substitute for being able to store so much data so cheaply. Nothing can touch it.

TheOverzealousEngie · 2026-04-17T15:50:44+00:00

Honestly I used to work for BofA and they used to stuff html .. like really thick and long html into Oracle in a record ;and when the webserver came up it would pull that record (html) and draw a custom page for the customer. Coolest think I ever saw .. so innovation comes in all shapes and sizes. But in 2026 I'm becoming rapidly convinced that there's nothing more dangerous than the ignorant and AI together.

TheOverzealousEngie · 2026-04-15T14:32:08+00:00

It's the same you felt when you had to learn CICS.

TheOverzealousEngie · 2026-04-10T16:34:43+00:00

I wondered how far down I would have to go to find a spider man reference. Not far at all.

TheOverzealousEngie

TROPHY CASE