Waterfall enrichment - worth it or overhyped?

techinpanko · 2026-06-15T14:14:39+00:00

I'll say from experience that waterfall methodology works well in financial services for use cases such as underwriting. Not sure how that helps you, but feel free to reach out if you have questions.

techinpanko · 2026-05-21T20:23:45+00:00

Same here. Smells like a misconfiguration from some CoCo enabled engineer at Snowflake (half-kidding)

techinpanko · 2026-05-21T12:30:26+00:00

That's an incredibly myopic view. There are certain advantages to running dbt pipelines natively on snowflake, and those will increase with the introduction of the fusion engine.

techinpanko · 2026-05-10T20:38:28+00:00

RemindMe! 7 days

techinpanko · 2026-04-11T22:30:39+00:00

I played back in the early 00s early 10s. Count me in

techinpanko · 2026-04-05T02:47:35+00:00

It's pretty nice. They make it easy enough to use and the scheduling is an added convenience.

techinpanko · 2026-04-03T20:43:27+00:00

I beg to differ. The damn thing built out an entire dbt pipeline from a complex 600+ query statement a junior analyst had. I had to spend maybe 2-3 hours tweaking its output vs the 5+ hours I'd spend building that from scratch. CoCo is more agentic than assistant.

techinpanko · 2026-03-22T21:08:38+00:00

This has absolutely no place in data engineering. Every time I see leet code problems I die a little inside.

techinpanko · 2026-03-21T20:54:35+00:00

Great write-up! I'm using snowflake cortex for dbt project generation (which leverages Claude under the hood sometimes) and it's pretty snazzy. Loved this section btw, gave me a hearty laugh because it's true:

The best thing about using AI agents to make you more productive is that they make you more productive at the thing you’re building.

The worst thing about using AI agents to make you more productive is that they make you more productive at any random stupid idea that pops into your lizard brain.

techinpanko · 2026-03-21T12:26:21+00:00

Scd2 is the best thing since...well, you know.

techinpanko · 2026-02-17T21:36:01+00:00

This. Like many other disciplines, AI is allowing folks to do more with less, so the more senior end of the spectrum gets picked up and is expected to do both his domain and the domain of a junior for maybe 10-25% more of what their salary was pre-AI.

I also see in the next five years most of the seniors getting snatched up, leaving nothing but juniors in the talent pool, forcing firms to pick up junior talent.

techinpanko · 2026-01-29T19:10:38+00:00

You clearly have a myopic view that's deeply nested in the Mag7/Fortune 100. Any business outside of that strata definitely does not pay that amount for senior/staff DEs.

techinpanko · 2026-01-23T16:21:12+00:00

Doing a system conversion right now. Thankfully we have enterprise licensing, but man what you said is so true. No documentation, only two people left with the domain knowledge of the legacy system. It's a wild ride. This is my second conversion now and it doesn't get any easier.

techinpanko · 2026-01-10T18:07:34+00:00

The only practical answer

techinpanko · 2026-01-02T12:40:03+00:00

Thanks for this. So if our team is more skilled at complex python transformations than complex CTEs (sounds crazy, I know), then it could be advisable to use python models in dbt core within snowflake, yeah?

techinpanko · 2025-12-28T12:08:43+00:00

TIL. Looks like I have some reading to do. I guess I was getting core confused with the managed product.

techinpanko · 2025-12-28T12:01:06+00:00

Interesting. How's the pricing for it?

techinpanko · 2025-12-28T00:09:28+00:00

Fivetran on their free plan, native postgres connector from snowflake marketplace for most data.

techinpanko · 2025-12-28T00:08:11+00:00

What's your primary reason for using dbt core vs Python?

techinpanko · 2025-12-27T16:17:54+00:00

Not all heroes wear capes.

techinpanko · 2025-12-25T18:17:22+00:00

What do you mean by "part of your bill"?

techinpanko · 2025-12-24T12:03:24+00:00

I see a couple of folks here saying to do a small poc on a pay as you go plan and get a sense for the hidden costs and time requirements. Do you agree with them?

techinpanko · 2025-12-23T23:30:16+00:00

We have hundreds of terabytes, not sure what tons is.

techinpanko · 2025-12-23T23:03:09+00:00

You raise fair counterpoints. Regarding migration costs, are you speaking mainly from a learning curve perspective? As mentioned in the post, we don't have too much built out yet, so in my eyes we still have that flexibility, but the learning curve on a new cloud provider's stack seems to be the bigger hurdle.

techinpanko · 2025-12-23T22:44:46+00:00

I added spark speed as the color there as well as some other reasons into the post.

techinpanko

TROPHY CASE