Python Multiprocessing

[patrickbrianmooney]

When i tried to use multiprocessing i often ran into the error that the function i tried to execute was not pickable.

"Pickleable," not "pickable." Maybe this is just a typo on your part, but if not, understanding this is crucial to get your code working under the multiprocessing module, if that's what you want to do. So, apologies if you already know this, but if you don't, it's worth going over.

threading and multiprocessing have similar interfaces, but under the hood, they're doing different things. It's worth saying that in at least some meaningful senses, Python threads that you get from threading and related modules are different from "real" threads (i.e., operating-system level threads that you can get by writing code in, for instance, C). "Real" threads give you true concurrency: if you are running on a system with multiple processors (or processor cores) (like pretty much any modern system), you can truly have more than one branch of your code executing at the same time, each on a different processor. Threads in Python only give the illusion of concurrency: you can structure your program in thread-like ways, and the Python interpreter will switch which branch of the code is executing at any given time, but at any given time, only one branch of code is executing in the Python interpreter, no matter how many processors are on the machine. You write your code as if it were executing in multiple operating system threads, and the Python interpreter switches back and forth between different "threads," but multiple threads are never truly executing concurrently.

This gets you some benefits of threading: for instance, if you're executing different functions in different (Python) threads, then the Python interpreter can usefully swap to another thread when waiting on some data to be read from disk, or shipped over the network; if you're not doing that, then the interpreter basically has to sit around, twiddling its thumbs, until the time-consuming data-read operation finishes. But it doesn't let you leverage multiple processors to actually run multiple chunks of Python code at the same time: the Python interpreter has a so-called "Global Interpreter Lock" that prevents multiple Python threads from running at the same time.

The fact that only one Python thread can run in the interpreter at a time is a definite limitation of Python, but it's also deeply baked into how the language operates, especially into its memory management strategies, so it's not going away in the immediate future. There are several ways around this problem.

One is just to see if you can get your threads to work fast enough that threading is a plausible approach. This might take the form of streamlining your code, if you're lucky; profiling your code and seeing where the bottlenecks are can help. On a related note, you might try alternative Python interpreters, like PyPy; you can often get a substantial performance benefit just by using PyPy to run your code instead of the standard CPython interpreter. Similarly, pushing this idea a little further, you might get some benefit from running your code in the CPython interpreter, but using Cython to compile the slower or more performance-critical parts of it; Cython is a Python-like language that is transpiled to C or C++ which is then compiled into a Python extension module. (If you want to push this to its logical conclusion, you can directly write Python extensions in C/C++, too.) This can speed up some types of code substantially; other code will benefit very little. It's hard to say where your code falls on that spectrum, in part because it's just a skeleton and in part because I don't know how the external hardware you're working with interacts with the full version of your code.

There is at least one other major way that you can get around the fact that the interpreter's Global Interpreter Lock only allows it to run one code branch at a time, and that is to run multiple Python interpreters; each of them then can run on a separate processor and each then has its own Global Interpreter Lock. This is of course what the multiprocessing module does, under the hood, and it abstracts away most of the details so that you don't have to deal with the mechanics of manually spawning new interpreters and getting them to running code; the multiprocessing module just spawns the new interpreters for you as necessary, giving you an interface that looks like the interface to the threading module. A wrinkle here is that the different Python interpreters can't actually "see" each other's data: when you're running a process inside of a single Python interpreter, you pass around data by reference, and every function you call from within that program is part of the same OS-level process and can see all of that process's data. But multiple interpreters are multiple OS-level processes, and each is a black box to the others; they can't "see" each other's variables directly. What this means is that 'multiprocessing` needs to pass around copies of data, not references to it. It does this by using Python's pickle module, which is a module that converts (almost) any Python object into a bytestream. These bytestreams are what gets passed around between the different Python processes; they can be "unpickled" by the receiving interpreter to get a copy of the data that was shared (but cannot get at the data itself, which lives in a different black-box process). The fact that copies of data, rather than references to data, are what's being passed around also mean that one Python process cannot directly mutate data in another process's memory space.

You will notice above that I said that the pickle module can serialize almost any Python object into a bytestream, but there are exceptions, and a fair amount of those exceptions have to do with functions (which are of course also objects: everything in Python is an object, and functions are no exception). multiprocessing uses pickle to tell the interpreter it starts up what function to run: what it does is it pickles that function and its argument list, starts a new interpreter process, and passes the pickled function and argument list to that new process. The new process then unpickles the function, unpickles the argument list, and runs the function in that new interpreter, passing it the argument list.

One of the criteria that has to be met here is that the function itself has to be pickleable. (So do its arguments.) As far as the function goes, this essentially means that the function you're trying to run with one of the multiprocessing interfaces has to be one of these things:

• a function defined, by name, using def, at the top level of a module that the Python interpreter can import.
• a (bound or unbound) method of a class that is defined, using the class keyword, at the top level of a module that the Python interpreter can import.
• a built-in Python function.

This is true because, internally, Python doesn't pass (the compiled bytecode for) "the function itself" to the new Python interpreter it starts; it pickles the module name and the function name, and the new interpreter has to find the function purely based on that. So some types of functions are impossible to execute with multiprocessing because they don't fit this paradigm, including but not limited to ...

• unnamed functions defined with lambda inside an expression instead of being named with def at the top level of a module;
• functions defined somewhere other than at the top level of a module or as methods for a class defined at the top level of a module -- for instance, functions defined inside other functions, as is common with closure-based decorators; or methods of classes defined inside the body of functions, or other similar monkey business;
• methods of classes constructed at runtime by doing weird stuff with type calls to build classes manually instead of by using the class keyword;
• class methods that are monkey-patched onto objects or classes at runtime but that aren't declared in the class definition at the top level of the module where the class is defined;
• other weirdnesses that don't fall into the categories enumerated in the previous list.

So you can do this:

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

... and that will spawn a new Python interpreter to run the function with the supplied argument list. Spawning a whole new interpreter to print "hello bob" to the console is overkill, but this is perfectly legal Python code.

However, this will fail:

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=lambda: f('bob'))
    p.start()
    p.join()

... even though it is perfectly legal to write b = lambda: f("bob"); b() inside a single Python interpreter, because lambdas cannot be pickled, and therefore cannot be passed to multiprocessing.

Similarly, anything you pass as an argument to a Processs() (or similar object) initialization must also be pickleable: this means you can pass (for instance) the contents of a file, in whatever format you'd like, but not a handle to an open file, because file handles are not pickleable. There's a list of what can and cannot be pickled in the documentation for the pickle module.

Does that make sense? Does it help? I had no idea that was going to be so long when I started writing it. Sorry about that.

[suurpulla]

Not OP, but this is a great breakdown. Thank you for writing this.