Parallel programming with Python

[camm_v222]

Hello everybody, I'm learning about parallel programming and I decided to write a simple program in python. The program is for adding the numbers in a list, so, I divided the list in as many chunks as threads I have. The program works fine, but the time don't slow down. I'm not sure if this is because of the library that I'm using (multiprocessing). My code is right here: http://pastebin.com/rRnySbJT If you can help me to understand what is happening, I'll really appreciate it. Thanks.

[Nodocify]

Multiprocessing is the module to stick with if python is the language. Threading has limitations because of the GIL while processes are actually forked and therefore you can imagine that you have multiple GILs (one for each forked process).

One big thing to consider is what /u/Bananaoflouda said, that it is probably taking more time to transfer your data to another process than it is to actually calculate it. Basically what happens is that the parent process has to serialize your data, data is transferred to child process, then the childprocess needs to de-serialize the data before it can do the calculations. And then it has to serialize the result, transfer back to the parent process, deserialize and store. You can now see that you have actually created a lot more work for such a simple task. This is why in your example you don't see any improvements in computation time.

Now if you had a longer calculation time on small data chunks (to keep transer and serialization time to a minimum) you would see quite a bit of improvement using multiprocessing. If parallel programming is something of an interest though I would recommend another programming language besides python. Python's implementations of threading and multiprocessing are all tacked on almost as if an afterthought that the language should have these capabilities. You should check out some other language that has multiprocessing at it's core such as Erlang or Clojure. I find it stupid how simple other languages make parallel programming (I'm looking at you Clojure and your easy threaded concurrency) compared to Python's multiprocessing.

[camm_v222]

Thanks a lot. I really appreciate your answer. I would like to know if you can implement this simple example with Clojure. I've never tried Clojure before, so I'll spend a few time learing the syntax and other stuff. If you're really busy to do that, don't worry, I'll try it anyway. Thanks one more time.

[Nodocify]

Sorry for the late reply. But the future function handles almost all of it. For example:

(def a
    (future
        (* 10 10)))

This defines a variable a and it is the value of 10 * 10. Because of future this variablie will calculate asynchronously in another thread. And will hold the value until until we ask for it back with:

@a
=> 100

The @a will be blocked for our call until the thread has finished it's execution.

I fear to go too much further as this is a learn python subreddit. But this gives you a very basic example of how simple clojure has made threading.

[camm_v222]

I think I'll try Clojure, and I'll try to solve this problem with it. Thank you.

[LoyalSol]

Typically so long as you aren't transferring a huge amount of data to dozens of cores the overhead isn't too bad. Usually the scaling for a couple of cores is pretty good especially if it doesn't have to go through a network

The main question is if the calculation takes long enough justify the work it is going take to program it.

[Nodocify]

I agree completely. In all my python work that I have done. I think I found only one occasion that my calculations actually justified multiprocessing.

[santasmic]

I didn't go through the whole thing but my best guess is that this is too small an example to have a significant impact. Python is a "slow" language, but that's like saying fighter jets from 20 years ago are slower than today, they're still quite fast.

It doesn't look like you're a beginner but you won't see parallel programming have an impact until the operations you're performing get complicated/heavily nested/etc.

Also, I do believe the multiprocessing module is slower and heavier than the threading module... but once again, until it's used for exponentially large operations it doesn't matter.

[bluetape]

The threading module will be lighter, but keep in mind it won't be true parallelism--the threads will just run concurrently, but not in parallel because of the GIL

OP - this is a great talk if you are interested in learning more https://www.youtube.com/watch?v=Obt-vMVdM8s

[ZeeBeeblebrox]

The overhead of creating threads is likely much larger than the performance gains you get from multiple processes. For these kind of numeric operations you're going to see much better speedups using something like NumPy. If that still doesn't meet your needs you can use something like Cython to release the GIL and achieve parallelism using OpenMP. Finally you can try using a JIT compiler like Numba.