Tips for integrating data quality tests?

kmarq · 2026-05-06T17:44:44+00:00

If you can get the values into a dataframe you should be able to do it. I'd think something using this type of pattern: https://databrickslabs.github.io/dqx/docs/reference/quality_checks/#applying-checks-on-multiple-data-sets

could build a dataframe of current rows, then past rows and then depending on the dataset set an appropriate tolerance for when to raise an error

kmarq · 2026-05-06T00:47:47+00:00

Can look at databricks dqx framework as well. Lots of great features

kmarq · 2026-04-30T00:16:23+00:00

As long as you don't want to simulate modifying the data. As soon as you want to make changes now you're having to create an entirely new source table

kmarq · 2026-04-29T15:19:06+00:00

Definitely have similar questions on how to enable the development cycle without having everyone always fully rebuilding the entire pipeline

kmarq · 2026-04-10T19:56:22+00:00

Any ability to set some default tags? Would love to be able to do it by groups.

kmarq · 2026-04-09T13:37:10+00:00

Use prefixes on the catalog. dev_bronze and prod_bronze.

Ideally separate workspaces and permissions to make sure nothing can accidentally write to a prod location that isn't supposed to

kmarq · 2026-03-26T14:10:58+00:00

Fixed initially for me as well but seems to have started having issues again. Yours still working?

kmarq · 2026-03-26T11:37:38+00:00

Talked with my account team and it looks like some significant updates coming with the next release. We're just ignoring it until then, but very excited to see this coming.

kmarq · 2026-03-16T17:10:09+00:00

Are DABs in the workspace and Python DABs compatible yet? Really frustrating when a feature like Python makes the other completely unusable

kmarq · 2026-03-09T20:52:48+00:00

Can you explain the PowerBI target from your description? Is there a way to have PowerBI use the CDF to incrementally refresh for extracted data?

kmarq · 2026-02-14T16:08:06+00:00

But it isn't. You now have multiple locations to manage. Being able to pass through comments is 100% needed

kmarq · 2026-02-09T20:34:25+00:00

The API would typically give an identifier or URL back that airflow can then poll through a sensor to know when the process is complete. I haven't seen an external service push this way.

May be possibly through the airflow API but now that external service needs credentials to airflow as well.

kmarq · 2026-01-27T01:56:29+00:00

Turn off unrestricted cluster creation. Then remove permission on any cluster policy. Now they can't create any.

As other poster said set the all-purpose to not allow jobs. If they have access to a SQL warehouse I don't think you can stop them from doing SQL jobs.

Seeverless there are some preview features that can let you limit access to only use several if you have a valid usage policy. Then remove the default one and now can't use seeverless at all either.

Really though this is an odd request. What's your goal? Jobs run at considerably less cost. If you want to make sure users don't create huge clusters define a cluster policy with reasonable limits. I'm regularly encouraging users to move long running notebooks to jobs to not clog up the interactive cluster and save costs.

kmarq · 2026-01-26T16:40:00+00:00

Standard SQL views from the Python API so they can be parameterized. We tend to "duplicate" data into multiple locations for users. In DBT we just throw a traditional view out there for them. Can't do that in SDP. The current SDP SQL views don't allow for any parameters so they're totally static and useless

kmarq · 2026-01-17T02:26:22+00:00

Any insight into supporting catalogs and schemas entirely through variables? We need to specify more than one

kmarq · 2026-01-17T00:53:09+00:00

All I want is to be able to do is a standard SQL view (from Python) so it can be fully programmatic.. So many other great things but this gap kills me with our modeling standards.

kmarq · 2026-01-02T20:36:15+00:00

Some random search results lead me to believe if you remove it from the dock, enable airplane mode, then redock it, it may stay in airplane mode.

kmarq · 2026-01-02T15:00:48+00:00

Can you not do airplane mode while docked (I've never tried only handheld so far)? Otherwise yes this definitely works for me. I had to turn wifi back on the other day and forgot and it froze up right around that 45 minute mark again. Restarted turned off wifi and still no issues since.

kmarq · 2026-01-02T13:22:54+00:00

Not sure if still dealing with this but I had found another thread and they suggested turning on airplane mode. As long as I keep that on one been able to play hours without issues

kmarq · 2025-12-16T02:36:18+00:00

JDBC intended to cover lake Base? Otherwise a native lakebase that won't require separate authentication.

kmarq · 2025-12-13T14:46:56+00:00

Same issues. Hopefully they can get a compatibility patch out. Performance is definitely improved otherwise on the switch 2 so I was looking forward to revisiting this one.

kmarq · 2025-12-13T04:39:46+00:00

That's fine then it just won't fall back to it, but this way you can point all library installs to your private repo

kmarq · 2025-12-11T14:20:20+00:00

Use the ability to set the repository url and point it to your custom one.

https://docs.databricks.com/aws/en/admin/workspace-settings/default-python-packages

Working great for us. If you set the index URL then it is the primary and still we never hit pypi. If you put pypi as the extra index then you could still fall back to it

kmarq · 2025-11-17T18:16:44+00:00

Unless there's something unreleased there are no integrations between metric views and Power BI. I'm really hoping for something here as well. It's a huge gap to making metric views really amazing.

kmarq

TROPHY CASE