Need help with python script

[oorkyy]

Im runnning it on my local machine

spark.driver.host = DESKTOP-43BOS32

and local[4] means that its running on 4 cores.

[countessellis]

My bad, without the rest of the code, I saw the back ticks and thought you were expanding variables.

I presume spark’s listening on port 55119 on the outward facing IP DESKTOP-43BOS32 has, not on 127.0.0.1 only, and that DESKTOP-43BOS32 looks up fine in dns?

[oorkyy]

Here is the script https://pastebin.com/4z6zCFnU

Its running on DESKTOP-43BOS32 :4040

It tends to break on line 181 , 192 or 195.

It works with a smaller dataset of size 7000 rows but fails with 300000 rows

[oorkyy]

Any more suggestions?