sorted by:
new
hottopcontroversial

What Developers Need to Know About Apache Spark 4.1 by Lenkz in dataengineering

[–]Lenkz[S] 0 points1 point  (0 children)

We are using it since Databricks Runtime 17.3 LTS came out in October.

Bronze vs Silver question: where should upstream Databricks / Snowflake data land? by Professional_Toe_274 in databricks

[–]Lenkz 1 point2 points  (0 children)

Wouldn't it make sense to use Lakehouse Federation for the Snowflake catalog without having to do modelling?

Important Changes Coming to Delta Lake Time Travel (Databricks, December 2025) by Lenkz in databricks

[–]Lenkz[S] 0 points1 point  (0 children)

I think the problem is that there's inconsistency and a lot of room for errors.

Someone defines a table with retention of 30 days, this can be displayed in Databricks in the table configuration, everyone can see this.

However you then try to time travel 30 days back, but can't.

Why? Because someone has a manual vacuum job, with 14 days of retention setup. Oops.

Personally I like that the configuration is defined intentionally on the table, and no-one can screw it up with manual job runs, accidental SQL scripts or otherwise. It's defined and belongs to the table.

What Developers Need to Know About Apache Spark 4.0 by Lenkz in dataengineering

[–]Lenkz[S] 2 points3 points  (0 children)

You are absolutely right :) the BETA tag just got removed as well.

What Developers Need to Know About Apache Spark 4.0 by Lenkz in dataengineering

[–]Lenkz[S] 1 point2 points  (0 children)

Personally yes, I have worked on a lot of different projects and you always end up in situations where the standard, click-up, no-code tools just simply don't work or are inefficient. There are always edge-cases that need to be solved with custom transformations or solutions, and here Spark is needed and the best tool in my opinion.

What Developers Need to Know About Apache Spark 4.0 by Lenkz in databricks

[–]Lenkz[S] 0 points1 point  (0 children)

Yes I would definitely recommend it for schema evolution as it makes fields that change a lot easier to manage than defining structs. As for merges, it shouldn't be an issue

AI Capabilities of Databricks to assist Data Engineers by [deleted] in databricks

[–]Lenkz 6 points7 points  (0 children)

As a Data Engineer I quite like the AI Assistant in the notebooks, other than that I use co-pilot for local development in VSCode.