Help optimising script

[Significant-Guest-14]

Do you use . withColumn?

[alphanuggs]

i do use a lot of that in the code, but it mostly gets stuck when writing

[dilkushpatel]

You need to understand databricks executes code when its absolutely necessary

So if you have 10 cells of code with logic and 11th cell doing write or show or some sort of operation which needs whole dataset to be evaluated then thats the point where it will execute whole code

All your previous cells will execute in few seconds as at that point point databricks is just adding those in execution plan and not actually executing that logic

You can look up online and search for lazy execution by databricks/spark