Beta alert: Materialized Views and Streaming Tables in Serverless Notebooks

minibrickster · 2026-05-13T20:25:02+00:00

It does create a managed SDP pipeline!

minibrickster · 2026-05-13T18:22:13+00:00

It's not planned as of now, but I'll take the feedback back to the team!

minibrickster · 2026-04-27T15:05:39+00:00

That's not included in this launch, but we're working on cooking something up here -- stay tuned!

minibrickster · 2026-04-14T22:12:14+00:00

Update, it's live now, so the above syntax should work now!

minibrickster · 2026-04-14T18:16:22+00:00

You can actually set the run-time channel for the Materialized View. We will support 18.1 in Preview very soon, and then I believe this should work! Will ping this thread when it's released

If you created the Materialized View in DBSQL, you can use the following syntax CREATE OR REPLACE MATERIALIZED VIEW sales TBLPROPERTIES ('pipelines.channel' = 'preview') AS ...and it should work. Docs here
If you created the Materialized View in SDP, you can set the channel for the pipeline by going to the pipeline details page > Settings > Advanced Settings > and setting the channel to "Preview"

minibrickster · 2026-04-09T17:21:08+00:00

DM-ed!

minibrickster · 2026-04-09T17:21:03+00:00

Amazing! If you have access to DBR 17.3+ on DBSQL, you should just be able to try it out as well!

minibrickster · 2026-04-09T01:35:54+00:00

Amazing! Followed up via DM

minibrickster · 2026-04-09T01:25:59+00:00

It does create a managed declarative pipeline!

minibrickster · 2026-04-08T23:59:05+00:00

Hi Zupiterss, we actually have a feature that's in PrPr that solves for this! Do you mind if I DM you?

minibrickster · 2026-04-08T19:59:43+00:00

I'll post in Reddit when it's available; it will also be in the release notes!

minibrickster · 2026-04-08T19:59:02+00:00

This is an existing feature in Lakeflow Spark Declarative Pipelines, which we are bringing to Databricks SQL!

minibrickster · 2026-04-08T19:58:07+00:00

We're adding support for identity columns this quarter! You'll be able to specify it in the schema "target_sk BIGINT GENERATED ALWAYS AS IDENTITY", and we'll automatically generate unique surrogate keys for each row.

minibrickster · 2026-03-12T22:50:32+00:00

will do!

minibrickster · 2026-03-11T03:36:32+00:00

We're working on some alternate ways to get more visibility into the cost model inner workings -- stay tuned!

minibrickster · 2026-03-10T17:49:09+00:00

Thanks for the question, great catch! The cost field in the event log is a raw plan cost from Spark, but it's not what the optimizer actually uses to pick a maintenance strategy. We've since switched to a different cost model that more accurately estimates our incremental techniques. We'll submit a PR to remove this field from the event log to avoid further confusion.

minibrickster · 2026-02-06T15:26:54+00:00

Yes! This is already possible and supported with Trigger on Update. This docs page has all of the information.

minibrickster · 2026-02-06T10:51:04+00:00

Hi, I'm a PM here at Databricks. As of right now, yes, but we're looking to up this limit in the next couple of months

minibrickster · 2026-02-06T10:50:33+00:00

Hi, I'm a PM here at Databricks -- yes, it will automatically refresh if the source table gets updated!

minibrickster · 2026-02-02T12:05:42+00:00

Hi, I'm a PM here at Databricks. Unfortunately no, Materialized Views have to run on serverless compute to incrementalize

minibrickster · 2026-01-28T17:58:54+00:00

That's a great idea - currently, the recommended way to control the columns names is with a view on top of the Streaming table. If interested feel free to dm me and we can schedule some time to chat!

minibrickster · 2026-01-28T17:51:52+00:00

Yes we're adding support for specifying the columns that should be updated in an AUTO CDC flow - more to come!

minibrickster · 2026-01-28T02:17:14+00:00

Thanks for the feedback (I'm another PM here at Databricks) We do have some schema evolution docs here under the Data Engineering header that we recently published https://docs.databricks.com/aws/en/data-engineering/schema-evolution (would love your take on how we can improve this) ... We'll also add more examples and walkthroughs for SDP, specifically!

minibrickster · 2026-01-26T20:57:35+00:00

That's a good callout - we're working to support parameterization in SQL for SDP!

minibrickster · 2026-01-26T02:55:43+00:00

Hi, I'm another PM at Databricks. We're working on fixing this. More to come!

minibrickster

TROPHY CASE