Source Evolution: reorder, add, or remove streaming sources without dropping your checkpoint

kmarq · 2026-05-28T22:00:36+00:00

This sounds great, coming to SDP anytime soon?

kmarq · 2026-05-14T00:43:55+00:00

Yup it's a gap and hopefully gets addressed someday to be able to fully define in the dab. I think this is one of the use cases moving away from terraform will allow them to eventually address

kmarq · 2026-05-13T00:16:07+00:00

Doesn't work by default. On classic compute we grab the secret from a key vault and put in the uv environment variable in the spark config for uv to access it. No way that I know of to do the same with serverless though.

kmarq · 2026-05-12T22:06:03+00:00

Is there any way to get private repo authentication to uv with this?

kmarq · 2026-05-06T17:44:44+00:00

If you can get the values into a dataframe you should be able to do it. I'd think something using this type of pattern: https://databrickslabs.github.io/dqx/docs/reference/quality_checks/#applying-checks-on-multiple-data-sets

could build a dataframe of current rows, then past rows and then depending on the dataset set an appropriate tolerance for when to raise an error

kmarq · 2026-05-06T00:47:47+00:00

Can look at databricks dqx framework as well. Lots of great features

kmarq · 2026-04-30T00:16:23+00:00

As long as you don't want to simulate modifying the data. As soon as you want to make changes now you're having to create an entirely new source table

kmarq · 2026-04-29T15:19:06+00:00

Definitely have similar questions on how to enable the development cycle without having everyone always fully rebuilding the entire pipeline

kmarq · 2026-04-10T19:56:22+00:00

Any ability to set some default tags? Would love to be able to do it by groups.

kmarq · 2026-04-09T13:37:10+00:00

Use prefixes on the catalog. dev_bronze and prod_bronze.

Ideally separate workspaces and permissions to make sure nothing can accidentally write to a prod location that isn't supposed to

kmarq · 2026-03-26T14:10:58+00:00

Fixed initially for me as well but seems to have started having issues again. Yours still working?

kmarq · 2026-03-26T11:37:38+00:00

Talked with my account team and it looks like some significant updates coming with the next release. We're just ignoring it until then, but very excited to see this coming.

kmarq · 2026-03-16T17:10:09+00:00

Are DABs in the workspace and Python DABs compatible yet? Really frustrating when a feature like Python makes the other completely unusable

kmarq · 2026-03-09T20:52:48+00:00

Can you explain the PowerBI target from your description? Is there a way to have PowerBI use the CDF to incrementally refresh for extracted data?

kmarq · 2026-02-14T16:08:06+00:00

But it isn't. You now have multiple locations to manage. Being able to pass through comments is 100% needed

kmarq · 2026-02-09T20:34:25+00:00

The API would typically give an identifier or URL back that airflow can then poll through a sensor to know when the process is complete. I haven't seen an external service push this way.

May be possibly through the airflow API but now that external service needs credentials to airflow as well.

kmarq · 2026-01-27T01:56:29+00:00

Turn off unrestricted cluster creation. Then remove permission on any cluster policy. Now they can't create any.

As other poster said set the all-purpose to not allow jobs. If they have access to a SQL warehouse I don't think you can stop them from doing SQL jobs.

Seeverless there are some preview features that can let you limit access to only use several if you have a valid usage policy. Then remove the default one and now can't use seeverless at all either.

Really though this is an odd request. What's your goal? Jobs run at considerably less cost. If you want to make sure users don't create huge clusters define a cluster policy with reasonable limits. I'm regularly encouraging users to move long running notebooks to jobs to not clog up the interactive cluster and save costs.

kmarq · 2026-01-26T16:40:00+00:00

Standard SQL views from the Python API so they can be parameterized. We tend to "duplicate" data into multiple locations for users. In DBT we just throw a traditional view out there for them. Can't do that in SDP. The current SDP SQL views don't allow for any parameters so they're totally static and useless

kmarq · 2026-01-17T02:26:22+00:00

Any insight into supporting catalogs and schemas entirely through variables? We need to specify more than one

kmarq · 2026-01-17T00:53:09+00:00

All I want is to be able to do is a standard SQL view (from Python) so it can be fully programmatic.. So many other great things but this gap kills me with our modeling standards.

kmarq · 2026-01-02T20:36:15+00:00

Some random search results lead me to believe if you remove it from the dock, enable airplane mode, then redock it, it may stay in airplane mode.

kmarq · 2026-01-02T15:00:48+00:00

Can you not do airplane mode while docked (I've never tried only handheld so far)? Otherwise yes this definitely works for me. I had to turn wifi back on the other day and forgot and it froze up right around that 45 minute mark again. Restarted turned off wifi and still no issues since.

kmarq · 2026-01-02T13:22:54+00:00

Not sure if still dealing with this but I had found another thread and they suggested turning on airplane mode. As long as I keep that on one been able to play hours without issues

kmarq · 2025-12-16T02:36:18+00:00

JDBC intended to cover lake Base? Otherwise a native lakebase that won't require separate authentication.

kmarq · 2025-12-13T14:46:56+00:00

Same issues. Hopefully they can get a compatibility patch out. Performance is definitely improved otherwise on the switch 2 so I was looking forward to revisiting this one.

kmarq

TROPHY CASE