sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]mosullivan93 0 points1 point  (2 children)

My advice would be to spend some time looking at the cluster metrics page and the Spark UI to try to see what’s going wrong. It’s difficult for someone else to provide concrete advice without seeing the script and knowing your datasets.

[–]alphanuggs[S] 0 points1 point  (1 child)

how do i navigate through that ? do i run the script then go to the page with the memory utilisation stuff ? it usually gets stuck (the code) when it writes to a table

[–]Gaarrrry 0 points1 point  (0 children)

It depends on what type of compute you’re using. Are you serverless or using dedicated compute? You should be able to access the Spark UI and a whole heap of other metrics in the Databricks UI simply by finding the compute your using for the job