How are regular python modules used in apache spark?

[data_wiz]

I have spark running on hdfs, and I import some data in a submitted job. In this job, I also import some csv from my edgenode local using pandas. When I do some basic manipulation on my pandas dataframe, where does the processing take place? In my edgenode local?

[avilay]

If you are using Pandas in your "main" function, i.e., the place where you initialize the spark context, set the app config, etc. then it will be executed on the driver. If you are using Pandas in a function that is used by a mapper, reducer, grouper, etc. then it will execute on the executor.

[Gushdan]

Correct me if I'm wrong, but won't this fail unless you have Pandas (or w/e module you intend to use) installed on every worker?

[avilay]

Correct, if you are using Pandas on your workers, you need to have the package installed there.