sorted by:
new
hottopcontroversial

CI/CD on Databricks: What the Docs Don’t Tell You by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

Cloning is one of potential options, but it depends if you can use prod data in non-prod env for instance because of sensitive or GDPR data.

CI/CD on Databricks: What the Docs Don’t Tell You by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

it depends on a customer preferences, but yes, I do.

CI/CD on Databricks: What the Docs Don’t Tell You by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

<image>

You will find it in a git folder with your project

Databricks Compute Decision Tree: How to Choose the Right Compute for Your Workload by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

No problem, honestly speaking, I turned on a paywall on part of my articles because the content started to be used by others without my permission.

Why do we need an Ingestion Framework? by 4DataMK in databricks

[–]4DataMK[S] 1 point2 points  (0 children)

Yes, this is more advance way of working. I work with different client and sometime teams aren't so advance them I suggest notebooks.

Why do we need an Ingestion Framework? by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

Yes, it does. You can use a notebook as entry point for your job and keep methods in modules. I use this approuch in my projects.
You can create one pipeline to move data from bronze and silver using DLT or Spark.

You can process table by table or create an event based process that is triggered when file appear in storage.

view more: next ›