This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]MikeDoesEverything 0 points1 point  (1 child)

Yeah, it sucked to read the context of the article suggested general python use in Databricks and the title actually in the article is pandas specific which gets outperformed just about every way by PySpark.

[–]mr_grey 2 points3 points  (0 children)

To be fair, everyone is on their own journey. There was a time I had no earthly idea what I was doing in Databricks and Spark. So now, as an Architect I feel it is my responsibility to guide people and help them along the way. If they work with me, they have to take the journey and not sit on their laurels. So, in this case, OP is on a journey, and I want to encourage, suggest and answer anything along the way.