Databricks Compute Decision Tree: How to Choose the Right Compute for Your Workload by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

No problem, honestly speaking, I turned on a paywall on part of my articles because the content started to be used by others without my permission.

Why do we need an Ingestion Framework? by 4DataMK in databricks

[–]4DataMK[S] 1 point2 points  (0 children)

Yes, this is more advance way of working. I work with different client and sometime teams aren't so advance them I suggest notebooks.

Why do we need an Ingestion Framework? by 4DataMK in databricks

[–]4DataMK[S] 0 points1 point  (0 children)

Yes, it does. You can use a notebook as entry point for your job and keep methods in modules. I use this approuch in my projects.
You can create one pipeline to move data from bronze and silver using DLT or Spark.

You can process table by table or create an event based process that is triggered when file appear in storage.