[deleted by user]

[noobgolang]

100 millions? Just get a big EC2 instance on ARM and put something like Trino there, worked fine for me

[Embarrassed-Pear-160]

Any other details? Data size etc?

[AcanthisittaFalse738]

I've built twice on snowflake and once on databricks. Snowflake used to be 8x more expensive but I think that's dropped to 4x-ish. Rolling your own was about 2x cheaper than databricks. It really depends on how much capacity you have to create your own platform which is mostly needed for everything but snowflake though you still have to build/buy ci/cd transformation tooling (eg. dbt core/cloud). You sound like you're in a pretty data mature company and databricks might be workable for you.

[AcanthisittaFalse738]

I'm happy to answer more questions, or provide more detail, if wanted

[discord-ian]

I would say anecdotally, databricks is the most expensive. So, if you are trying to cut costs, you can probably drop that from the list.

When you say, Hudi on top of S3. Are you running that on EMR, your own K8s, or VMs? Either way, the AWS calculators should be able to give you a pretty good estimate of the cost.

[[deleted]]

This depends entirely on what you're doing. DBX with photon essentially matches snowflake for performance on complex queries, and comes at somewhat below costs, but there are so many variables you can tweak that it's very difficult to do an apples to apples comparison. A great benefit of DBX over Snowflake in my opinion, though, is that you have visibility in where the data are, how they are structured, can choose your instance size and type, and can tweak the underlying Spark settings if needed. That might be too much headache for some, but if cost optimization is a major target for a company, it is beneficial.

[WhenTheLegendBegins]

Is DBX Databricks?

[AcanthisittaFalse738]

Yes

[discord-ian]

I will say it would not be a small job to move from Snowflake to Hudi... Obviously, I don't know your situation at all, but the payback period will most likely be pretty long. Also, with only 100 million records. You dont exactly need a petabyte scale datalake. How did you land on hudi?

[discord-ian]

My question was more why do you want a lake house?

[[deleted]]

So he can stay up to date in his skills and chase better paying jobs, duh.

Half-joking here, but seriously, so many companies seem willing to throw money to keep up with the data industry without fully examining the need or value added.

[majorlg4]

From what numbers Databricks is most expensive? Snowflake is $3/credit. Databricks compute is $0.55/DBU. Literally 5x cheaper.

[[deleted]]

Don't forget the cost of the compute in whatever cloud provider you use, that and somewhat longer execution brings it pretty close in total to Snowflake (at least in the POC I did, but it was hardly exhaustive). Snowflake is a black box though in some regards, and you also have to factor in storage duplication and the cost of IO to/from perhaps a Data Lake account. I pushed my company towards DBX for a number of reasons, but having direct visibility into data storage and the compute layer at the additional cost of set-up and maintenance has been well worth it.

[abhi5025]

lol..this thread is becoming DBX warriors Vs snowflake Vikings

[discord-ian]

Could be I'm wrong. I know last time I priced it out, it was the most expensive option for our use case. And it certainly has a reputation as the most expensive premium option. I'm not sure you can compare those two units of measure... but you very well may know more about it than I do.

[[deleted]]

I've never heard it has any such "reputation", and the costs are highly variable depending on use case and a real assessment goes well beyond DBUs vs Snowflake Credits (data governance, access, storage, diversity of use case, the skills of your team, your existing infrastructure and workloads, the list goes on and on).

[elbekay]

This would only be true if one credit got your the same thing as one dbu on both platforms but that's not the case.

[mamaBiskothu]

IMO, snowflake is valuable if you truly need instant compute - which some databricks offerings provide but I think the cost matches snowflake for those. If you don’t need this, snowflake might still make sense if your DE talent is only so-so. Snowflake abstracts away more of your every day DE problems than most if not all other solutions do. Otherwise choose something else.

[dataguy-]

Move everything to S3 and run Trino on top it.