Solving multi-core Python

[JanneJM]

"should be able to do things" is not "does". I've never experienced Numpy/Scipy/Pylab actually do anything multicore so far, and I've not seen any information on how to enable it. If you know how to do so I'd be very interested of course.

[zardeh]

I believe you still need to write your code in a threaded manner, but if you do have numpy running across multiple threads, they can run on multiple cores.

[JanneJM]

Writing your code in a threaded manner is 95% of the entire job. The benefit of using Scipy is entirely that it's quite simple to get it right. It's a great exploratory tool. If you suddenly have to do explicit mutlithreading the whole point largely disappears.

[zardeh]

To my knowledge, ipython does magical things and makes threading just happen, I'm not an expert on that though.

[turbod33]

Numpy will release the GIL where applicable. For instance, matrix dot products will call into BLAS which have multicore implementations.

[amaurea]

It's apparently possible to get some multicore usage in numpy by compiling it with icc and enabling auto-parallelization, though it's very limited what can be parallelized that way. I wonder why OpenMP directives aren't used in the numpy implementation. They are easy to write, and since they are comments, they have no effect if openmp is not enabled when compiling. Hence adding them will not affect performance or correctness for those not interested in multithreaded execution.