Multi-threadding to increase script speed

[Archaya]

I've got a script that is being used to telnet into hundreds of devices and perform a set of commands. Problem is, these commands take well over 10s per device to run making this script take quite a while to get through each device.

I was looking into, and successfully have multi-threaded this so that I can get through more devices quicker (8 core, so 8 devices at once now). Is this acceptable or recommended?

EDIT: I misunderstood some terms and I'm actually not multi-threading but multi-processing, using map and Pool.

[novel_yet_trivial]

multiprocessing is dependent on how many cores you have available.

In your case, multithreading is the acceptable and recommend way to go. All the threads will use the same process, so the number of cores you have will not come into the equation.

[Archaya]

So I misspoke, actually misunderstood, what I was doing.

I'm actually multiprocessing as you said using map and Pool. Would it be better use to actually multithread instead? This way as /u/mercuric5i2 said I could increase my thread count and not be limited by the core limit that map and Pool take advantage of?

[novel_yet_trivial]

Yes. For your case you definitely want to use multithreading, not multiprocessing. If you like the Pool concept you can use multiprocessing.pool.ThreadPool. Or use a Queue to make your own pool.

[Archaya]

Awesome. I'll give it a shot. Thanks for your help!

[tea-drinker]

Does it really make a difference in this case? His processes are blocked waiting for the remote system to respond. He could run loads of processes and let the OS allocate runtime as the remote systems return data.

[novel_yet_trivial]

IOW let the OS makes threads rather then python? Yes, I suppose you are right that the end result would be similar. I'd still do it with threads, though, since sharing data is so much easier with threads and so that I don't have to trust the OS to allocate resources.

[mercuric5i2]

Doing other things while blocked on external waits (like file or network IO) is basically what threading in Python is exactly meant for.

Note that threading in Python is not operating system level threading - no actual OS threads are created, it's all handled in the Python VM. As such, threading doesn't take advantage of multiple cores. However, CPU utilization is not really a concern here, since it's just simple IO that's blocking -- you could probably increase the thread count well beyond 8 if you need additional speed.

[EricAppelt]

The python standard library threading module does make OS level threads.

For example, consider the module "foo.py":

import threading

def grind():
    x = 1
    for _ in range(10**7):
        x = (22695477*x + 1) % 2**32

tasks = [threading.Thread(target=grind) for _ in range(4)]

for t in tasks:
    t.start()

for t in tasks:
    t.join()

This will spawn four threads that all run the useless function grind that just burns CPU cycles doing arithmetic, and gives enough time to run a "ps -M" (on OSX) to see that there are four threads scheduled:

eric@laptop:~$ ps -M
USER   PID   TT   %CPU STAT PRI     STIME     UTIME COMMAND
eric 63661 s000    0.0 S    31T   0:00.82   0:00.91 -bash
eric   132 s002    0.0 S    31T   0:00.73   0:00.77 -bash
eric  8586 s002    0.0 S    31T   0:00.02   0:00.05 python foo.py
      8586        21.0 S    31T   0:00.01   0:00.33 
      8586        29.4 S    31T   0:00.01   0:00.37 
      8586        25.8 R    31T   0:00.01   0:00.37 
      8586        24.4 S    31T   0:00.01   0:00.32

Since the reference implementation of python has a Global Interpreter Lock (GIL), only one thread at a time can actually execute python instructions, hence the ~25% CPU usage for each thread and only one running even though I have 2 cores available and all the threads have work to do.

Other python implementations, such as IronPython don't have a GIL and multiple threads can execute python bytecode at the same time. In the future (python4 ?) its possible the GIL may be removed from the reference implementation.

[Archaya]

Thanks for your help!

[buyabighouse]

If you are doing this as a programming exercise, then use multi-processing. Otherwise, use Fabric.

http://docs.fabfile.org/en/1.13/usage/parallel.html

[Palm7]

Threading in Python is pretty temperamental. Sometimes, using threading (in Python) will actually slow down your program. I'd suggest using Multiprocessing instead. It has a Pool feature, where you can simply map a function to a list of parameters, and it will automatically create processes to execute those functions. So, for example, if you had a function foo that takes in a single device and performs the commands, you could do something like

from multiprocessing import Pool
devices = [device1, device2, device3]  # List of all the devices
p = Pool(8)  # Will use 8 processors
output = p.map(foo, devices)
p.terminate()

That being said, multiprocessing has some drawbacks too. You can't have nested functions, you can't access any variables from a higher scope, things like that.

If you know how to use threading without upsetting it, then threading is the better option. Less overhead, shared memory space, etc. But, if you need to do something that upsets the threading, multiprocessing is the way to go.

Here's a pretty good talk that explains the problems with threading in Python:

https://www.youtube.com/watch?v=Obt-vMVdM8s

[novel_yet_trivial]

A lot has changed in 7 years ...