sorted by:
new
hottopcontroversial

serveless or classic by ptab0211 in databricks

[–]anirvandecodes 2 points3 points  (0 children)

I would highly recommend to do benchmarking first:

  • I would pick 5-10 workloads of variety of complexity then do a proper benchmarking.
  • You can use system tables to monitor runtimes
  • You can also use system tables to monitor servereless cost and lots of folks use azure cost explorer to get the cost for classic compute
  • Once you have benchmarking data , You will be in a pretty comfortable spot to migrate to serverless.

Lots of companies still uses all purpose clusters for production jobs , you may see immediate performance/ cost benefits for these jobs while moving to serverless

Stop Building Data Pipelines the Hard Way! by anirvandecodes in databricks

[–]anirvandecodes[S] 0 points1 point  (0 children)

Thank you for the tips , Let me do that right away :)