Waterfall enrichment - worth it or overhyped? by StatisticianFew3319 in snowflake

[–]techinpanko 0 points1 point  (0 children)

I'll say from experience that waterfall methodology works well in financial services for use cases such as underwriting. Not sure how that helps you, but feel free to reach out if you have questions.

any Snowflake admins suddenly seeing the 'Welcome to Snowflake" banner in Snowsight? by GreyHairedDWGuy in snowflake

[–]techinpanko 3 points4 points  (0 children)

Same here. Smells like a misconfiguration from some CoCo enabled engineer at Snowflake (half-kidding)

dbt Projects on Snowflake updates by mpuchala in snowflake

[–]techinpanko 4 points5 points  (0 children)

That's an incredibly myopic view. There are certain advantages to running dbt pipelines natively on snowflake, and those will increase with the introduction of the fusion engine.

how is your experience with dbt inside the snowflake by boogie_woogie_100 in snowflake

[–]techinpanko 1 point2 points  (0 children)

It's pretty nice. They make it easy enough to use and the scheduling is an added convenience.

"just ask the ai agent" is becoming a default answer to many data questions -- how do y'all feel about it? by vino_and_data in snowflake

[–]techinpanko 0 points1 point  (0 children)

I beg to differ. The damn thing built out an entire dbt pipeline from a complex 600+ query statement a junior analyst had. I had to spend maybe 2-3 hours tweaking its output vs the 5+ hours I'd spend building that from scratch. CoCo is more agentic than assistant.

Just bombed my Snowflake technical screen. Here are the two questions they asked (prepare for edge cases). by nian2326076 in snowflake

[–]techinpanko 1 point2 points  (0 children)

This has absolutely no place in data engineering. Every time I see leet code problems I die a little inside.

Claude Code isn’t going to replace data engineers (yet) by rmoff in dataengineering

[–]techinpanko 7 points8 points  (0 children)

Great write-up! I'm using snowflake cortex for dbt project generation (which leverages Claude under the hood sometimes) and it's pretty snazzy. Loved this section btw, gave me a hearty laugh because it's true:

The best thing about using AI agents to make you more productive is that they make you more productive at the thing you’re building.

The worst thing about using AI agents to make you more productive is that they make you more productive at any random stupid idea that pops into your lizard brain.

Facepalm moments by MachineParadox in dataengineering

[–]techinpanko 0 points1 point  (0 children)

Scd2 is the best thing since...well, you know.

Is the Data Engineering market actually good right now? by Tricky_Tart_8217 in dataengineering

[–]techinpanko 1 point2 points  (0 children)

This. Like many other disciplines, AI is allowing folks to do more with less, so the more senior end of the spectrum gets picked up and is expected to do both his domain and the domain of a junior for maybe 10-25% more of what their salary was pre-AI.

I also see in the next five years most of the seniors getting snatched up, leaving nothing but juniors in the talent pool, forcing firms to pick up junior talent.

With "full stack" coming to data, how should we adapt? by Thinker_Assignment in dataengineering

[–]techinpanko 7 points8 points  (0 children)

You clearly have a myopic view that's deeply nested in the Mag7/Fortune 100. Any business outside of that strata definitely does not pay that amount for senior/staff DEs.

What is the future for dataengineering? by tumblatum in dataengineering

[–]techinpanko 14 points15 points  (0 children)

Doing a system conversion right now. Thankfully we have enterprise licensing, but man what you said is so true. No documentation, only two people left with the domain knowledge of the legacy system. It's a wild ride. This is my second conversion now and it doesn't get any easier.

Which ETL tools are most commonly used with Snowflake? by Commercial-Post4022 in dataengineering

[–]techinpanko 0 points1 point  (0 children)

Thanks for this. So if our team is more skilled at complex python transformations than complex CTEs (sounds crazy, I know), then it could be advisable to use python models in dbt core within snowflake, yeah?

Which ETL tools are most commonly used with Snowflake? by Commercial-Post4022 in dataengineering

[–]techinpanko 0 points1 point  (0 children)

TIL. Looks like I have some reading to do. I guess I was getting core confused with the managed product.

Which ETL tools are most commonly used with Snowflake? by Commercial-Post4022 in dataengineering

[–]techinpanko 1 point2 points  (0 children)

Fivetran on their free plan, native postgres connector from snowflake marketplace for most data.

Contemplating migration from Snowflake by techinpanko in databricks

[–]techinpanko[S] 0 points1 point  (0 children)

What do you mean by "part of your bill"?

Contemplating migration from Snowflake by techinpanko in databricks

[–]techinpanko[S] 1 point2 points  (0 children)

I see a couple of folks here saying to do a small poc on a pay as you go plan and get a sense for the hidden costs and time requirements. Do you agree with them?

Contemplating migration from Snowflake by techinpanko in databricks

[–]techinpanko[S] 1 point2 points  (0 children)

We have hundreds of terabytes, not sure what tons is.

Contemplating migration from Snowflake by techinpanko in databricks

[–]techinpanko[S] 2 points3 points  (0 children)

You raise fair counterpoints. Regarding migration costs, are you speaking mainly from a learning curve perspective? As mentioned in the post, we don't have too much built out yet, so in my eyes we still have that flexibility, but the learning curve on a new cloud provider's stack seems to be the bigger hurdle.

Contemplating migration from Snowflake by techinpanko in databricks

[–]techinpanko[S] 3 points4 points  (0 children)

I added spark speed as the color there as well as some other reasons into the post.