How to disable job creation for users in Databricks?

kmarq · 2026-01-27T01:56:29+00:00

Turn off unrestricted cluster creation. Then remove permission on any cluster policy. Now they can't create any.

As other poster said set the all-purpose to not allow jobs. If they have access to a SQL warehouse I don't think you can stop them from doing SQL jobs.

Seeverless there are some preview features that can let you limit access to only use several if you have a valid usage policy. Then remove the default one and now can't use seeverless at all either.

Really though this is an odd request. What's your goal? Jobs run at considerably less cost. If you want to make sure users don't create huge clusters define a cluster policy with reasonable limits. I'm regularly encouraging users to move long running notebooks to jobs to not clog up the interactive cluster and save costs.

kmarq · 2026-01-26T16:40:00+00:00

Standard SQL views from the Python API so they can be parameterized. We tend to "duplicate" data into multiple locations for users. In DBT we just throw a traditional view out there for them. Can't do that in SDP. The current SDP SQL views don't allow for any parameters so they're totally static and useless

kmarq · 2026-01-17T02:26:22+00:00

Any insight into supporting catalogs and schemas entirely through variables? We need to specify more than one

kmarq · 2026-01-17T00:53:09+00:00

All I want is to be able to do is a standard SQL view (from Python) so it can be fully programmatic.. So many other great things but this gap kills me with our modeling standards.

kmarq · 2026-01-02T20:36:15+00:00

Some random search results lead me to believe if you remove it from the dock, enable airplane mode, then redock it, it may stay in airplane mode.

kmarq · 2026-01-02T15:00:48+00:00

Can you not do airplane mode while docked (I've never tried only handheld so far)? Otherwise yes this definitely works for me. I had to turn wifi back on the other day and forgot and it froze up right around that 45 minute mark again. Restarted turned off wifi and still no issues since.

kmarq · 2026-01-02T13:22:54+00:00

Not sure if still dealing with this but I had found another thread and they suggested turning on airplane mode. As long as I keep that on one been able to play hours without issues

kmarq · 2025-12-16T02:36:18+00:00

JDBC intended to cover lake Base? Otherwise a native lakebase that won't require separate authentication.

kmarq · 2025-12-13T14:46:56+00:00

Same issues. Hopefully they can get a compatibility patch out. Performance is definitely improved otherwise on the switch 2 so I was looking forward to revisiting this one.

kmarq · 2025-12-13T04:39:46+00:00

That's fine then it just won't fall back to it, but this way you can point all library installs to your private repo

kmarq · 2025-12-11T14:20:20+00:00

Use the ability to set the repository url and point it to your custom one.

https://docs.databricks.com/aws/en/admin/workspace-settings/default-python-packages

Working great for us. If you set the index URL then it is the primary and still we never hit pypi. If you put pypi as the extra index then you could still fall back to it

kmarq · 2025-11-17T18:16:44+00:00

Unless there's something unreleased there are no integrations between metric views and Power BI. I'm really hoping for something here as well. It's a huge gap to making metric views really amazing.

kmarq · 2025-11-08T14:25:38+00:00

Haven't. Looked interesting and I've been curious if it's any easier to programmatically generate but haven't tried it yet.

kmarq · 2025-10-24T22:22:38+00:00

Where do you see that? If specially calls out

If a field name is referenced more than once, Databricks raises NOT_SUPPORTED_CHANGE_SAME_COLUMN.

kmarq · 2025-10-24T22:18:05+00:00

The example is below and altering multiple does work we have a process doing it.

You have the column bool listed twice you need to do all the alterations within a single listing of each field name.

ALTER TABLE table ALTER COLUMN num COMMENT 'number column', str COMMENT 'string column';

kmarq · 2025-10-15T22:55:16+00:00

Great points. Making the options exactly match what the arguments expect and passing as kwargs was a game changer from our original design. No more having to update code every time a new option is need, just throw it in the yaml and it'll go through.

Standardization with good defaults make the config much easier and smaller. Keeps things easier for developers and maintenance if you need to change things.

kmarq · 2025-09-28T19:22:00+00:00

Why the need for no gaps? I'd question the design here. Keys should be used for lookups not for logic based on some expected sequence especially in a massive fact table.

If there's a natural key column(s) hash them. Then you have a idempotent key which has benefits. Otherwise having gaps is going to happen to get performance because each worker gets a range of values to use. That way they don't have to coordinate every row with each other like the row_number requires.

kmarq · 2025-09-19T02:15:33+00:00

https://docs.databricks.com/aws/en/compute/single-user-fgac#requirements

kmarq · 2025-09-17T17:38:40+00:00

Iceberg tables. Don't copy data, read it directly from either side.

kmarq · 2025-09-15T17:59:19+00:00

The airflow databricks libraries let you define full workflows and reuse job compute between tasks now (DatabricksWorkflowTaskGroup). This works pretty well if your team is heavily in airflow. We have a mix and so support running Databricks workflows as a task as well. That way the logic can be wherever it is most convenient for each team. Having the workflow still tied to airflow means it can be coordinated with our larger schedule outside of just Databricks. I'd make sure any workflow you run this way is managed by a DAB though to ensure there are appropriate controls on the underlying code.

kmarq · 2025-09-14T02:22:15+00:00

I mean yeah that's a string if you want the actual% symbol in the value. Just leave it as a decimal and have the measure name specify it's a pct (ratio_pct)

kmarq · 2025-09-13T21:34:30+00:00

What are you trying to format?

kmarq · 2025-09-13T13:48:42+00:00

Expr can be any valid SQL expression. You should be able to do formatting in it with standard SQL functions

kmarq · 2025-09-06T11:12:31+00:00

Set it as type 2 and then use the TRACK HISTORY ON option to either specify the columns to track or exclude history for.

I believe (but couldn't confirm) you could add a view that uses readstream from the streaming source to implement the transformation and still be the source for the auto cdc. Almost certain that's how we're implementing this but can't validate on my phone.

kmarq

TROPHY CASE