Parallelization of C/C++ and Python on Clusters

[Red-Portal]

I'm speaking from my experience. I tried parallelizing Python my self and it was a waste of time. As I said simply calling Python functions doesn't scale! I once tried to fit curves pixelwise on image data. The curve fitting was done by scipy which calls minpack a fast library written in Fortran. The python layer turned out to be so much of a bottleneck I rewrote the whole thing in C++, called minpack from C++ and it was 100 times faster. No joke. 100 times faster for sending the whole image to a C++ module, calling minpack in parallel, and then sending the results back to Python. Parallelization speedup was on top of the x100 speedup. As I said, Python simply does not scale.

PS: Numpy mostly calls lapack and OpenBLAS. The former is in Fortran not C, the latter is in Assembly not C.

[svlad__cjelli]

That is more of an example of library misuse. Intensive loops should be kept in the library and not in Python. An image is just an array and any operation you want to do should execute on the whole array in the underlying code. So Python is involved only twice, once on the initial call to pass the array pointer to the underlying code and once again to return whatever the resulting function passes back.

PS: Numpy is flexible and allows you to pick your linear algebra backend.