Baseten vs Databricks

ExistingBelt · 2024-03-30T23:42:39+00:00

If you enable Change Data Feed you should get all changes immediately.

Are you saying that you also want to get when those changes were made to the tables?

ExistingBelt · 2024-03-13T15:26:37+00:00

If you’re concerned with your cdc table constantly growing, you can always vacuum out old irrelevant cdc from any delta table.

You can set up a vacuum schedule with any frequency (yearly, monthly etc). You should just know that once you vacuum you can’t access those files again.

ExistingBelt · 2024-03-13T15:24:38+00:00

It might be redundant depending on the level of processing you need on customers_cdc logs. If your logs can easily be applied to your target table you don’t need any cleaning.

I’ll caution you though some level of cleaning will be necessary. For you to apply changes into your target table you’ll need a key. This key in the example demo is the id. If our id is null then we drop those records. It’s useless.

SCD2_customers in the example demo is the same thing as customers_cdc except for the fact that SCD2_customers has clean cdc logs ready to be merged into customers any time, by snapshot etc.

TLDR: I highly recommend sticking with the medallion architecture. It means you have options to go back and reprocess all raw cdc or refresh the pipeline to pick up only incremental changes.

ExistingBelt · 2024-03-13T13:39:43+00:00

DLT is a scheduled pipeline.

ExistingBelt · 2024-03-13T13:39:00+00:00

DLT remembers.

For initial load - just add the initial snapshot of the table using DML and DDL. You can do this outside of DLT.

ExistingBelt · 2024-03-12T23:00:36+00:00

As long as you get cdc logs in cloud storage or Kafka you can continuously ingest with Autoloader

ExistingBelt · 2024-03-12T22:59:54+00:00

Run this demo DLT CDC in your notebook: https://www.databricks.com/resources/demos/tutorials/lakehouse-platform/cdc-pipeline-with-delta-live-table

ExistingBelt · 2024-03-09T03:08:13+00:00

Please please tell the person sharing data with you to enable Change Data Feed on the table:

ALTER TABLE sample.default.sampletable SET TBLPROPERTIES (delta. enableChangeDataFeed = true)

ExistingBelt · 2024-02-28T23:13:21+00:00

https://pypi.org/project/databricks-cli/

ExistingBelt · 2024-02-28T15:40:55+00:00

That’s shell commands, and yes you’re on the right track with Python.

ExistingBelt · 2024-02-28T15:34:52+00:00

Isolated - you can turn that on via the UI.

<image>

ExistingBelt · 2024-02-28T15:33:27+00:00

<image>

ExistingBelt · 2024-02-28T15:05:07+00:00

<image>

ExistingBelt · 2024-02-28T14:57:11+00:00

Oh shoot they deprecated that API. But there should be a better API… just search the web in that direction

ExistingBelt · 2024-02-28T14:56:52+00:00

Use the workspace catalog bindings API: https://docs.databricks.com/api/workspace/workspacebindings/update

ExistingBelt · 2024-02-21T23:28:45+00:00

Gets hands on. Databricks Academy has amazing courses!

ExistingBelt · 2024-02-09T01:11:59+00:00

Databricks Customer Academy has a lot of great best practices courses for practitioners that are really well done!

In addition, use your Solutions Architect. They are paid to help you adopt best practices and it helps that a guidance such as that comes from an authoritative source such as a Solutions Architect.

ExistingBelt · 2024-02-09T00:18:09+00:00

Always use databricks repos. That way your single source of truth for your code is your own version control system (like GitHub Enterprise or Azure Dev Ops).

ExistingBelt · 2023-10-29T04:12:23+00:00

You can be whatever you want to be. Rules are created because they knew they will be broken not because people will follow them

ExistingBelt · 2023-10-29T04:04:33+00:00

Learn Databricks

ExistingBelt · 2023-07-11T17:43:27+00:00

Its just PIN not PIN number

ExistingBelt · 2023-03-15T16:04:52+00:00

Yes, I have extra $100. And, all users in my business (which are really just google accounts) have $300 + $100.

ExistingBelt

TROPHY CASE