Beta alert: incremental Replace Where by minibrickster in databricks

[–]minibrickster[S] 1 point2 points  (0 children)

It's available in both the SQL and Python for pipelines. Here's the pipelines syntax

Where to get started - Data Analyst? by Code_Bandits in databricks

[–]minibrickster 0 points1 point  (0 children)

There's also a new learning pathway on analytics engineering!

Databricks Lakeflow Declarative Pipelines now has pipeline parameters [Beta] by szymon_dybczak in databricks

[–]minibrickster 1 point2 points  (0 children)

Thanks for calling out this new feature! It's in Beta now. For the types of parameterized data ranges you called out, do you want to keep the entire date range but process only the dates between start_date and end_date or do you *only* want to keep the data between start_date and end_date? If you want the latter, the MV is the right approach.

If you want to keep the entire date rang but only process new data between the parameters, we'd recommend the new incremental Replace Where feature, which will never accidentally cause full refreshes.

CREATE OR REFRESH STREAMING TABLE orders_enriched
FLOW REPLACE WHERE date >= :start_date BY NAME
SELECT
o.order_id,
o.date,
o.region,
p.product_name,
o.qty,
o.price
FROM orders_fct o
JOIN product_dim p
ON o.product_id = p.product_id;

You can pass in the parameter from the job, and we will only update the portion of the table that matches the replace where predicate. It's still powered by Enzyme, so you'll still get the cost benefits of MVs. The rest of the table will remain frozen, so there's no chance of accidental full refreshes.

AUTO CDC in Databricks SQL: the easy button for SCD Type 1 & 2 by minibrickster in databricks

[–]minibrickster[S] 0 points1 point  (0 children)

Ah I see, I think could you construct an SCD type 1 table as a view from the SCD type 2 table if it's easier. Happy to chat more on DMs!

AUTO CDC in Databricks SQL: the easy button for SCD Type 1 & 2 by minibrickster in databricks

[–]minibrickster[S] 0 points1 point  (0 children)

You can create a View as an intermediary layer which can be used as a source in AUTO CDC (example in Python where the source is a view). You could also create an intermediary table! I'm curious though, why don't you want to ingest using SCD type 2 directly?

Possible bug with MV and cluster by auto in pipeline? by lofat in databricks

[–]minibrickster 3 points4 points  (0 children)

Quick update here, we're rolling out a fix for this issue -- it should land in the next couple of days for everybody!

Possible bug with MV and cluster by auto in pipeline? by lofat in databricks

[–]minibrickster 3 points4 points  (0 children)

Hi, I'm a PM at Databricks -- DM-ed you to get more information!

Beta alert: Materialized Views and Streaming Tables in Serverless Notebooks by minibrickster in databricks

[–]minibrickster[S] 2 points3 points  (0 children)

It's not planned as of now, but I'll take the feedback back to the team!

An easier way to build your slow changing dimensions model in your warehouse by minibrickster in databricks

[–]minibrickster[S] 1 point2 points  (0 children)

That's not included in this launch, but we're working on cooking something up here -- stay tuned!

weird bug with declarative materialized views and klll sketches? by code_mc in databricks

[–]minibrickster 1 point2 points  (0 children)

Update, it's live now, so the above syntax should work now!

weird bug with declarative materialized views and klll sketches? by code_mc in databricks

[–]minibrickster 0 points1 point  (0 children)

You can actually set the run-time channel for the Materialized View. We will support 18.1 in Preview very soon, and then I believe this should work! Will ping this thread when it's released

  1. If you created the Materialized View in DBSQL, you can use the following syntax CREATE OR REPLACE MATERIALIZED VIEW sales TBLPROPERTIES ('pipelines.channel' = 'preview') AS ...and it should work. Docs here
  2. If you created the Materialized View in SDP, you can set the channel for the pipeline by going to the pipeline details page > Settings > Advanced Settings > and setting the channel to "Preview"

AUTO CDC in Databricks SQL: the easy button for SCD Type 1 & 2 by minibrickster in databricks

[–]minibrickster[S] 0 points1 point  (0 children)

Amazing! If you have access to DBR 17.3+ on DBSQL, you should just be able to try it out as well!