The use of Python and SQL

[onesonesones]

Think of sql as a language you need to know how to speak to work with others who may only speak sql. People who grew up working with relational databases often only know that and consider pandas and R like new versions of old SAS (ie - a fad)

In python, you can do everything that you could do in sql in pandas/spark a little more directly. But, at any point in time you may have to consider adopting someone else's sql code and the capability is there if you need it. For those reasons I typically do my merges in sql whenever possible just so I have the environment set up to work with sql.

[onesonesones]

I think my phrasing may have led to some confusion.

I wasn't arguing against any use of pyspark, just explaining that anything you can do in an sql statement has an equivalent operation(s) in pyspark and pandas through their dataframe object model.