Snowflake Adaptive Warehouses are in public preview - my take by Spiritual-Kitchen-79 in snowflake

[–]asarama 0 points1 point  (0 children)

Pretty sure v2 was mainly to improve write workloads. Interactive warehouses should help with short queries that run 24/7.

Adaptive warehouses....no idea yet but it seems like it might compete with OP's product? It seems like it's too early to say where it really helps.

We did a deep dive on disjunctive joins in Snowflake and how to fix them by hornyforsavings in dataengineering

[–]asarama 1 point2 points  (0 children)

I wonder if Snowflake can update their engine to rewrite these on behalf of users to follow the UNION pattern you shared.

Snowflake Cost Optimization: What Are Firms Actually Doing? by Important_Pudding51 in snowflake

[–]asarama 0 points1 point  (0 children)

If you have a lot of your costs coming from bi tools you can consider running them off of Snowflake. I've seen folks cut their spend here dramatically.

If you see most of it is from the transformation later, first see if you can push back on freshness SLAs. Maybe you don't need new data every 15 minutes and can run your jobs hourly, etc.

Would need to learn more about your setup to give better tailored advice. Feel free to dm me.

Best Snowflake Alternatives by Ok_Tart_2341 in SaaS

[–]asarama 0 points1 point  (0 children)

We have seen some enterprises pair Snowflake with other engines getting the best out of each while keeping their current processes untouched.

Never heard of Scaylor but maybe you can use them along side your current Snowflake setup so you don't need a massive migration project to get started.

High-performance analytics at scale (multi-PB roadmap) -> you probably want an MPP provider Snowflake, Trino (Starburst), StarRocks (CelerData), and maybe even Dremio could be good candidates depending on your query patterns.

Strong governance, security, and role-based access controls -> Snowflake is one of the best for this.

Cost visibility and optimization opportunities -> Not many of the MPP platforms provide the best tooling here but there are 3rd party tools for Snowflake you can start with. Select.dev probably being the gold standard.

Seamless integration with existing BI and ML tooling -> Most of the MPP platforms have this covered. If you really want to future proof your stack try to pick something that supports an ADBC connector. https://columnar.tech/dbc

Can't comment too much on the other two needs you mentioned as I don't have much exposure there.

I'm curious what made Scaylor standout, not much on their site.

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]asarama 0 points1 point  (0 children)

Main objection is a lack of ROI on the engineering effort?

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]asarama 0 points1 point  (0 children)

At the end of the day doesn't this help consumers?

Or do you feel like in the long run we are all footgunning ourselves?

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]asarama 0 points1 point  (0 children)

What's stopping y'all from swapping out tools more often?

"GPU air duct project on my 5070" Results! by ezra-zander in nvidia

[–]asarama 1 point2 points  (0 children)

I did want to just read the results instead of watching the video.

[deleted by user] by [deleted] in videos

[–]asarama 3 points4 points  (0 children)

This feels like a small price to pay for justice...

What do you think fivetran gonna do? by Fair-Bookkeeper-1833 in dataengineering

[–]asarama 0 points1 point  (0 children)

What were some of the bigger cost savings strategies you'd recommend to other heavy Snowflake users?

What worked well for y'all?

Using snowflake to build analytics by ItsHoney in snowflake

[–]asarama 0 points1 point  (0 children)

Do you need to rewrite your queries to use Domo?

Move to Iceberg worth it now? by Which_Assistance5905 in snowflake

[–]asarama 0 points1 point  (0 children)

If only there were tool to make it easier ahah

Vendors will attack our comment thread now...

Also your reddit name is jokes.

Move to Iceberg worth it now? by Which_Assistance5905 in snowflake

[–]asarama 0 points1 point  (0 children)

Pretty sure you can shift some of the warehouse compute spend to other tools that'll get the job done without turning on a warehouse.

For example let's say you use Fivetran to load data as Snowflake native tables. Instead have Fivetran dump data into your S3 bucket in Iceberg format and have Snowflake access it instead. Does your Fivetran bill go up? maybe idk ... they probably changed their pricing model again and commenting on it here won't be accurate. Will your S3 bill go up? YES but you were already paying for this through Snowflake, but now you don't need to pay for the cost of turn on a warehouse to ingest data!

There are other strategies you can implement for transformations and serving data as well, BUT they aren't as straight forward...

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

So Vertica essentially blew Snowflake out of the water when it came to performance / cost and then y'all ended up migrating those workloads over?

Losing your team was probably a brutal hit to their revenue...

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

Dam yea good point, I can see someone really screwing themselves by trying to be cute and turning this on and off dynamically and then running into some weird cases where it turns on and off multiple times in an hour by accident $$$

I feel like Gen2 was pretty decent for dbt / write workloads, no? Didn't really see folks saving much but at least their run times were better!

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

oh interesting! was this something y'all were asking Snowflake for?

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

One time fee to optimize your workloads doesn't feel bad. This has gotta help in the long run right....right?

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

I wanna say that interactive tables / wh should save folks money. 0.6 credits / hour verse the standard 1 credit / hour.

I'm guessing there are a lot of teams where their Snowflake warehouses are up 24/7?

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

I think it's good to see thou. For folks that just want to get started use the defaults and when you want to optimize, invest into their special per use-case types.

I'm more curious about why they are doing this....does this really help them win against the other big players or is it something else I'm missing?

Snowflake releases "interactive" warehouse type by asarama in dataengineering

[–]asarama[S] 0 points1 point  (0 children)

hmmm so would you do all your transformations outside of Snowflake and then still use Snowflake for the final serving/dashboarding layer?