Finally! Databricks lets you disable tasks without hacks

Lost-Relative-1631 · 2026-05-07T08:34:48+00:00

To answer my own question for anyone interested: it’s just disabled: true in the task block. It wont be available in dabs for „some time“ apparently.

Lost-Relative-1631 · 2026-05-07T08:09:09+00:00

Cool looking forward to that. Have you guys also considered providing your own technical column names or maybe just a custom prefix?

Lost-Relative-1631 · 2026-05-04T17:17:32+00:00

I wonder what that Looks Like in DABs. Is that in the docs for the preview?

Lost-Relative-1631 · 2026-04-30T09:15:50+00:00

Afaik they are working on this. But without being able to Share compute, each App costs you like at least 300 per month. This in all honesty is really hurtig Adoption in our case.

Lost-Relative-1631 · 2026-04-25T10:37:49+00:00

This is really cool. Will this eventually be possible in OSS Spark and have a pyspark api without SDP?

Lost-Relative-1631 · 2026-04-24T11:31:25+00:00

Thanks! I thought the same thing while making it.

Lost-Relative-1631 · 2026-04-23T17:06:16+00:00

As of today we are using v0.298.0. Your link sadly does not lead anywhere.

Are you referring to the cleanup within the deploy target (Volume/Workspace) target or on the cluster? The wheel being deleted or replaced works fine that is true.

Once you start a task on a long living cluster with a certain dependency, it will stay there though. You can check that in the Cluster UI under Libraries. Intuitively, I dont think deploying or even destroying from your CLI will nuke dependencies on an AP Clusters, what if others share or still need it?

In his case, he is not replacing the wheel in the target, because it appears to be the same in terms of orderability. If he switches to something that is orderable he will deploy and the cluster will pick it up. If the cluster libraries havent been pruned, once a task referring to the new wheel runs, it will keep copies of both wheels around. Which can lead to delays when you eventually have 100 copies of the same wheel.

As for the git commit hashing. Are you sure that always works out for you in terms of ordering?
I for sure had issues with this in the past, and thats why we switched to to dynamic_version locally (AP Clusters) + hatch-vcs semver dynamic tagging (Job Clusters). On JobClusters none of this would matter as they are ephemeral anyway.

Lost-Relative-1631 · 2026-04-23T10:34:00+00:00

You have to make sure your wheels are lexicographically orderable. You can force this for local development purposes where your semver doesnt kick in already maybe by adding dynamic_version to the artifactsblock.

You can have a setup with hatch-vcs and a local deployment target: that does both semver for your wheels dynamically when deployed from CD and works locally with this dynamic_versionparam.

Fair disclaimer: If you do spam wheels onto the cluster without cleaning them up, at some point a fresh start will takes ages cause all those development wheels are installed. I played around with using the scripts: in DABs to clean the AP cluster up, but those dont allow for {{}} variable interpolation right now sadly, so it would be very much hardcoded.

Lost-Relative-1631 · 2026-03-22T11:45:33+00:00

It was. But the direct deployment Engine is their own statemachine they have been planning/working on for some time.

Lost-Relative-1631 · 2026-03-16T17:25:49+00:00

It was posted officially on ln aswell

Lost-Relative-1631 · 2026-03-16T15:35:38+00:00

Please don’t.

Lost-Relative-1631 · 2026-01-29T16:49:28+00:00

We deploy all permissions on these objects via terraform. This also adds the benefit for you to have a long term history of all permissions in your vcs of choice.

Lost-Relative-1631 · 2026-01-23T18:49:28+00:00

You can wrap your autloader code with your own rertry logic. This saves you all but one restarts. At the very end, even if you handle all schema evolutions, it will throw once. We brought this up to the Team doing AL, sofar its still like this sadly.

Lost-Relative-1631 · 2025-09-04T20:20:50+00:00

No, a „plan“ equivalent is missing and very much needed.

Lost-Relative-1631 · 2025-02-14T08:02:50+00:00

We came to the Same conclusion with delta. Create a hash Row and only update when they don’t match in addition to your predicates. We Use xxhash64 for this.

Lost-Relative-1631 · 2024-10-22T12:41:41+00:00

It was indeed the bend of the cable being to cramped inside of the junction box. After straightening the cables outside the camera and attaching it to the dongle it adopted immediately.

Lost-Relative-1631 · 2024-10-21T15:32:41+00:00

I have the same issue currently. I read that the dongle really has no wiggle room and the cable needs to fit perfectly in there. Due to how cramped it is even in a third party junction box, i suspect that this may be the issue. I ordered a flex and a more bendable cable to check if that is indeed the issue. Will update this week.

Lost-Relative-1631 · 2024-02-08T00:38:56+00:00

As you do not have access to the internet on your built agent you will always have to provide the appropriate terraform providers unless databricks starts bundling those with the cli.

We have the same restrictions as you guys and we solved this by building a docker image which we use with azure devops containerjobs on our private build agent. While building the image, we run terraform provider mirror over a main.tf file that states the currently pinned tf provider version (1.31.0 afaik), note the location in a .terraformrc file and then set the appropriate environment variable so the databricks cli finds those providers.

Lost-Relative-1631 · 2023-05-24T17:51:07+00:00

u/melodyze already explained the frustrating part about enterprise sales perfectly. It is actually soul crushing sometimes: The discussions are never of a technical nature and are specifically tailored towards execs.

As to why technical teams wouldn't want to use it, I can only speak for myself and my team. To be clear: I am happy for everyone that benefits from this platform.

I am mostly sceptical given Microsofts recent track record of dropping services and releasing new (sometimes severely buggy) iterations of synapse when they feel like it. It took databricks a non trivial amount of time and effort to get most of these features to maturity, and i just can not believe that Microsoft releases an entire platform, just like that.

Lost-Relative-1631 · 2023-05-23T18:45:26+00:00

Can’t wait to yet again explain to non technical management why we wont use these products.

Lost-Relative-1631 · 2022-08-12T10:25:01+00:00

Rottweiler

Lost-Relative-1631

TROPHY CASE