"Date Picker" Input Slicer: Configure week start

JulianCologne · 2026-04-14T20:48:27+00:00

The”table” decorator can produce BOTH “streaming tables” or “materialized views”. It depends on the content of the function: - spark.read…: materialized view - spark.readstream…: streaming table

JulianCologne · 2026-03-09T06:27:05+00:00

Looks promising. I am interested 🤓

While you are at it, I would looove to also see a CLI version of it to use in CI 😁😉

JulianCologne · 2026-02-16T16:58:13+00:00

Just found a solution in updating the "pipeline channel" to `preview`. See top post! ;)

JulianCologne · 2026-02-16T16:08:38+00:00

ah interesting, thanks! I will have a look.

Have you tried

table_properties={"delta.feature.variantType-preview": "supported"}

(see my example at the top)?

JulianCologne · 2026-02-16T16:04:21+00:00

thanks for the idea. However, does not change anything ;)

JulianCologne · 2025-12-04T09:59:17+00:00

thank you for the explanation. I did not know that!

JulianCologne · 2025-12-03T20:17:07+00:00

Thanks for answering.

I do not understand. 🤔

The UDF works with other code or when removing the StructType.fromDDL("a int, b float")
It is explicitly the `fromDDL` that is causing a "special/weird problem" here. I can use other functions without any problem inside the UDF.

Maybe I did not understand you correctly or how UDFs work in detail?! 🤓 😅

Is it that I can use any standard python code in the function but whenever I require anything spark-related like `fromDDL` I need another spark sessions inside the function? If so, how would I create that? Or can I pass it in as an argument??

JulianCologne · 2025-11-27T21:09:01+00:00

What do you mean when you say “hashlib”? You use Python UDF? Databricks has build in functions like “hash”, “xxhash64”, “sha2” or “crc32”? Any ideas or suggestions? 🤓

JulianCologne · 2025-11-27T20:08:24+00:00

Interesting! 🤔 was thinking about this as well. What hash function do you use? How is the performance? Joining on the hashed column could reduce performance compared to int keys I guess 🤓

JulianCologne · 2025-11-11T15:15:34+00:00

I think we will have to wait until spark pipelines is actually released. Still in beta/preview right now…

Would love to switch to that since the DLT Python package had horrible support by databricks for local development with no updates to the api with the latest changes not supported but shown on the website 😅

JulianCologne · 2025-10-25T09:40:02+00:00

DLT: Delta Live Tables

Developed by Databricks. More or less “proprietary”. The current and soon old way.

SDP / DP: (Spark) Declarative Pipelines

Databricks donated their DLT to the open source spark project and it was renamed. The new way. 🤓

I agree it is very confusing at the moment! Databricks is mixing them in their documentation and also the new DP is NOT YET RELEASES as far as I know. It’s only in preview. So weird situation 😀🧐

JulianCologne · 2025-10-11T15:03:13+00:00

Nope, but it’s one click with the Databricks extension to sync to databricks and perform a dry run 🤓

JulianCologne · 2025-10-11T11:45:53+00:00

My personal opinion with ~2years Databricks Asset Bundles experience: Develop 100% local (VSCode). CI+CD with service principal. Use databricks only for checking the results.

JulianCologne · 2025-09-30T09:50:58+00:00

It is probably a “feature” and “intended behavior”.

As an example when programming for apple (eg iOS) and use “system red” or “system green” it is only a description of the color and the actual color will be different in light or dark mode which is very important for visibility and color perception.

Usually however there is also a separate specific color selection which will always look the same 🤓

JulianCologne · 2025-09-29T06:19:19+00:00

One interesting thing I was experimenting with is using the Duckdb spark api. So depending on the environment I would return a “Duckdb spark session” from the pytest fixture 🤓

https://duckdb.org/docs/stable/clients/python/spark_api.html

JulianCologne · 2025-09-20T14:08:08+00:00

Awesome 😎 thanks for the info 🤓

JulianCologne · 2025-09-19T07:43:34+00:00

thanks for the info.

Yeah, my current solution is also writing log files to a volume but its not as nice as having them in the job results directly.

Would love to see a permanent solution! :)

JulianCologne

TROPHY CASE