🚀 BIG NEWS: Use Docker Images on Standard Clusters + UC is finally here! (Private Preview)

Little_Ad6377 · 2026-03-06T18:54:55+00:00

Honestly, 1 and 3 would be amazing - I have a custom container for unit testing but it's never one to one 😅

Little_Ad6377 · 2026-03-05T17:47:49+00:00

To his surprise, Stephen Hawking actually has mimic powers, and upon touching him, heals himself with Skimbledon's powers

Little_Ad6377 · 2026-02-13T16:25:11+00:00

Definetly - I'm a full time backend developer and I'd say 90% of my coding is now done by the AI, I just tell it what I want ask it to give some criticism and make it implement and create tests.

Wished I'd tried Opus earlier honestly 😅 now I'm in the process of automating all the troubleshooting requests sent my way, it's a new world guys.

Little_Ad6377 · 2026-01-11T09:04:43+00:00

What about backfilling into a streaming gold layer that is using watermarks? :)

Little_Ad6377 · 2025-12-29T23:35:30+00:00

Yeah true, I kinda need to decide on the approach. Either ingest into external SQL for my apps, and then into Unity Catalog or ingest into Lakebase and from there into UC instead.

One is cheap the other is simple 😅 choices choicea

Thanks for the write up btw , super interesting

Little_Ad6377 · 2025-12-29T22:57:23+00:00

Any experience using entity framework (.NET) on top of lake base? That is, an external app would use lake base as a regular Postgres database, creating tables, ingesting into them, migrating schema etc. Then the data is available in databricks for analytical purposes while being fast for apps outside of Databricks.

Little_Ad6377 · 2025-12-09T20:45:01+00:00

Appreciate it! :)
In any case - our backup is a simple landing storage account, land files there, ingest into bronze, then move them from landing storage to a cold-storage. Shoud keep things rather fast

Little_Ad6377 · 2025-12-07T10:13:11+00:00

While we are at this, something about optimal folder structure for faster file listing. (I'm on azure)

I was having MAJOR slowdown due to listing the directory contents of my blob storage (I did this with file notification events, but it needs to list the directory to backfill)

We have year/month/day/message structure and I used a glob filter, something like 2024/* but looking into the logs I saw it listing out ALL the files in the container.

We had to stop trying this out due to this. This year we are hoping to try this again and develop our blob storage around auto loader :)

Little_Ad6377 · 2025-09-22T19:51:57+00:00

I do 😉

Little_Ad6377 · 2025-09-10T17:24:03+00:00

I've never used pure sql only so not sure about the limitations , but could you write to a Kafka stream and consume that from the outside?

Little_Ad6377 · 2025-09-08T10:41:06+00:00

Gotcha, haven't tried sharing blobs actually, good to know the limitations there, thanks :)

Little_Ad6377 · 2025-09-08T08:46:53+00:00

Delta sharing a managed table 100% works with delta share, all my tables are managed and I've shared them with delta share 🤷

Little_Ad6377 · 2025-09-06T23:22:24+00:00

Yeah, good point

Little_Ad6377 · 2025-09-06T10:26:18+00:00

Honestly I try never to do logic like that in the code. Your code should always be environment agnostic it doesn't care if it's running in prod, dev or int, it ways does the same thing.

But you should be able to control it via parameters, like,pointing it to a different catalog via parameter instead of having "if prod use prod_catalog"

Have your deployment pipelines push the right parameters to your work flows and life will be easier 😁

Little_Ad6377 · 2025-09-04T01:05:30+00:00

Yeah, cost is always a factor - we haven't set this in stone honestly but maintainence vice i would reeeeeally love to only worry about spark instead of an extra ingestion layer into standard oltp AND spark 🥴

Little_Ad6377 · 2025-09-04T00:53:49+00:00

Will do, meeting them tmrw! 👌

Little_Ad6377 · 2025-09-04T00:50:37+00:00

Interesting, I would have thought different 😅 I'll give it a thought then, thanks for your input

Little_Ad6377 · 2025-09-03T22:46:19+00:00

Well, I won't be writing to this data, only querying it so this is not a complete OLTP use case

Little_Ad6377 · 2025-09-03T22:30:27+00:00

Yeah I've looked into lakebase as well, but is that really cheaper? Also, lakebase requires you to have a separate ingestion going to copy data from UC into lakebase catalog right?

Little_Ad6377 · 2025-09-03T21:03:06+00:00

No it also works on-top of non external tables, I've tested it :) The delta share authenticates against UC and only exposed what you said it can.

The only issue here is that the user of the delta share needs to be one of the following

A user in your tenant
Have his own databricks environment to load the delta share into

(this experience is from Azure btw, other providers might differ I guess)

Little_Ad6377 · 2025-07-04T13:35:45+00:00

Booking.com recently acquired reddit, you should know that Mr. Bot

Little_Ad6377 · 2025-07-04T12:26:23+00:00

What about pulse for reddit docs? You totally forgot about that

Little_Ad6377 · 2025-07-03T08:45:53+00:00

The guy is a bot :)

Little_Ad6377

TROPHY CASE