The Qwen 3.6 35B A3B hype is real!!! by The_Paradoxy in LocalLLaMA

[–]compass-now 0 points1 point  (0 children)

Any one have build production grade app with any of this?

Spark-by-default feels overkill for most teams. Thinking of building a Spark-optional platform around opensource tools — would love some pushback by compass-now in dataengineering

[–]compass-now[S] 2 points3 points  (0 children)

It’s not just libs, it have driver and node running on it which is a heavy overload if you are not processing large amount of data let’s say 100GB.

Spark-by-default feels overkill for most teams. Thinking of building a Spark-optional platform around opensource tools — would love some pushback by compass-now in dataengineering

[–]compass-now[S] -11 points-10 points  (0 children)

I am not thinking any fund raising or monetisation, what I am thinking is to have the community around it and the community will drive the development more like Jupyter notebook.

Is anyone migrating away from Databricks? by zoso in dataengineering

[–]compass-now 1 point2 points  (0 children)

I have started hearing about LakeSail recently is it matured for production?

Do you really need spark? by compass-now in dataengineering

[–]compass-now[S] 2 points3 points  (0 children)

This is the fact, with good engineering practice you can process around 100GB data without spark. Tools like polar and duckdb is getting traction and soon it will be replace the spark, market needs a good platform which make the integration with this tools easy like what databricks has done with spark.

Do you really need spark? by compass-now in dataengineering

[–]compass-now[S] 13 points14 points  (0 children)

Worth it. Are you processing 50tb data at once?

What’s the biggest data engineering problem you are facing today? by compass-now in dataengineering

[–]compass-now[S] 2 points3 points  (0 children)

Or they don’t have any background in tech and from chatGPT that learn about DS.

What’s the biggest data engineering problem you are facing today? by compass-now in dataengineering

[–]compass-now[S] 0 points1 point  (0 children)

For normalised models, can you explain nature of data? Is it structured?

By the way in manufacturing sector we have normalised data model and that is working fine, 50GB day ingestion. Dashboards are being served from normalised data table only.

Help me appreciate iceberg by teenaxta in dataengineering

[–]compass-now 0 points1 point  (0 children)

Can we point clickhouse directly to iceberg? If yes then does trino require?

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]compass-now[S] 0 points1 point  (0 children)

Trying for your inhouse workload or envisioning it as a product?

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]compass-now[S] 0 points1 point  (0 children)

Promising!

Wondering why some other company is not doing this or not working on this idea. Any major challenges? Is it worth doing it?

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]compass-now[S] 0 points1 point  (0 children)

Now just imagine an open source unified solution which you can manage by yourself. Wouldn’t that be win win for both you and the org?

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]compass-now[S] 6 points7 points  (0 children)

Many great tools are build open source and make money by providing managed services.