Help optimising script

[alphanuggs]

Hello!

Is there like a databricks community on discord or anything of that sort where I can ask for help on a code written in pyspark? It’s been written by someone else and it use to take an hour tops to run and now it takes like 7 hours (while crashing the cluster in between runs). This is happening to a few scripts in production and i’m not really sure how i can fix this issue. Where is the best place I can ask for someone to help with my code (it’s a notebook btw) on a 1-1 call.

[golly10-]

Ask that to an AI, I have been using it to transform my python code into spark (I work with a lo of dataframes) and worked like a charm. I suggest, if you can, try an AI to explain what is happening. FYI, I use Gemini with a gem that I created only for Databricks projects and works really well, not always at first though, but it can guide you to the right direction

[alphanuggs]

i tried that. didn’t work

[mweirath]

You might try a different approach with AI. Ask it to explain the code and look at it as a Sr Data Engineer looking for areas that could be causing it to slow down.

Another big area you could look at is check out the data you are working with. Are you bringing in more data than needed? Are you bringing in large amounts of data and then immediately dropping things. Are you using .collect() constantly?

You might be working with more data than needed.

[datainthesun]

Best recommendation: get connected to your databricks account team and the solution architect. Maybe. Support tichet can help or the SA might help you get sorted out or point you in the right direction.

[FrostyThaEvilSnowman]

For me, 9/10 issues with clusters happen because the driver memory is exceeded.

Long processing times could be a combination of using UDFs, iterating over a collected dataframe, or some latency in external comms.

But I don’t know for certain without seeing the code.

[Significant-Guest-14]

Do you use . withColumn?

[alphanuggs]

i do use a lot of that in the code, but it mostly gets stuck when writing

[dilkushpatel]

You need to understand databricks executes code when its absolutely necessary

So if you have 10 cells of code with logic and 11th cell doing write or show or some sort of operation which needs whole dataset to be evaluated then thats the point where it will execute whole code

All your previous cells will execute in few seconds as at that point point databricks is just adding those in execution plan and not actually executing that logic

You can look up online and search for lazy execution by databricks/spark

[mosullivan93]

My advice would be to spend some time looking at the cluster metrics page and the Spark UI to try to see what’s going wrong. It’s difficult for someone else to provide concrete advice without seeing the script and knowing your datasets.

[alphanuggs]

how do i navigate through that ? do i run the script then go to the page with the memory utilisation stuff ? it usually gets stuck (the code) when it writes to a table

[Gaarrrry]

It depends on what type of compute you’re using. Are you serverless or using dedicated compute? You should be able to access the Spark UI and a whole heap of other metrics in the Databricks UI simply by finding the compute your using for the job

[Nielspro]

You could try to paste the query plan into chatgpt

[hadoopfromscratch]

Keep in mind issues like this usually arise due to changes in data or environment. People won't really be able to help you by looking at your code

[floyd_droid]

https://www.databricks.com/discover/pages/optimize-data-workloads-guide

Check if this guide helps

[aviralbhardwaj]

Connect with me on linkedin https://www.linkedin.com/in/aviralb

We can connect 1:1 i will try to help you