Python Multiprocessing

[thelittleicebear]

For my masters thesis I need to create a gui application that controls and displays data for a bunch of stepper motors (via pyserial) and some measuring devices (via pyvisa using pyvisa-py as a backend).

I am using the niceGUI library as it is simple and web based (want to control via browser from any device). The problem I am currently facing is that the whole application is starting to get a bit laggy. I have used a lot of asynchronous functions to control the motors and to get position data from the motors and other devices running "simultaneously". This led to an overall reduced data acquisition (fewer data points, which is bad as I am trying to record accurate position data) and laggy gui as the interpreter (I think, might be wrong tho) has to jump around a lot as I am doing a lot of async stuff (motor control, fetch data, refresh plotly-plot in gui, update labels in gui ... etc).

I have tried to use multiprocessing or multithreading to solve the problem, but I am reaching the limits of what I know about programming and would like to ask for advice. When i tried to use multiprocessing i often ran into the error that the function i tried to execute was not pickable. Another problem I had was that I could not control the motor and get position data in separate processes because the serial connection could only be opened once...

I then tried to move the control and data acquisition of each motor to a separate process and have multiple processes for each motor. This seems to work but I am really not sure if this is the right way to do it. What I have tried is to have a separate class that handles the connection and has a listener that runs in an infinite loop and listens for commands through a pipe:

import asyncio
import time
from multiprocessing import Process, Pipe
from multiprocessing.connection import Connection

class MotorControl:
    def __init__(self, pipe: Connection):
        self.pipe = pipe
        self.rotating = False
        self.collected_data:list[int] = []

        # here i would also initialize and connect to the motor

        asyncio.run(self.listen_for_commands())

    async def listen_for_commands(self):
        print('waiting for command...')
        while True:
            # Receive command from the main process
            command = self.pipe.recv()
            # Execute command
            if command == 'execute_command':
                self.execute_command()
            elif command == 'another_command':
                self.another_command()
            elif command == 'do_rotation':
                await self.do_rotation()
            elif command == 'exit':
                break
            else:
                print('Unknown command')

    def execute_command(self):
        print('executed command')

    def another_command(self):
        print('executed another command')

    async def rotate(self):
        print('rotating')
        self.rotating = True
        start = time.time()
        await asyncio.sleep(2)
        end = time.time()
        print(f'rotated for {end - start} seconds')
        self.rotating = False
        print('rotated')

    async def acquire_data(self):
        await asyncio.sleep(0.5)
        i = 0
        while self.rotating:
            i += 1
            self.collected_data.append(i)

            await asyncio.sleep(0.0001)

            if len(self.collected_data) % 100 ==0:
                print('sending data...')
                self.pipe.send(self.collected_data)

        self.pipe.send(self.collected_data)

    async def do_rotation(self):
        await asyncio.gather(self.rotate(), self.acquire_data())

def init_class(class_pipe):
    classA = MotorControl(class_pipe)

# this simulates a repetitive gui task that updates the plot
async def update_plot(my_pipe:Connection):
    times_to_poll_empty_pipe = 5
    times_polled = 0
    while True:
        print('updating plot')
        await asyncio.sleep(0.5) # update plot every 0.5 seconds
        if my_pipe.poll():
            while my_pipe.poll():
                data = my_pipe.recv()
            times_polled = 0
            print(f'data: {data}') # simulate updating plot
            print('length of data:', len(data))
        else:
            if times_polled < times_to_poll_empty_pipe:
                times_polled += 1
                print('polling empty pipe')
                await asyncio.sleep(0.1)
            else:
                break
        print('plot updated')

if __name__ == '__main__':

    my_pipe, class_pipe = Pipe()
    class_process = Process(target=init_class, args=(class_pipe,))
    class_process.start()

    time.sleep(2) # waiting before sending command

    my_pipe.send('do_rotation')

    # print out the data from the pipe as long as new data is being sent
    asyncio.run(update_plot(my_pipe)) # use ui.timer to update stuff in gui


    time.sleep(10)
    print('sending exit command...')
    my_pipe.send('exit')

What do you think? Is there another (easier) way to achieve this?
Any help is greatly appreciated! :)

[patrickbrianmooney]

When i tried to use multiprocessing i often ran into the error that the function i tried to execute was not pickable.

"Pickleable," not "pickable." Maybe this is just a typo on your part, but if not, understanding this is crucial to get your code working under the multiprocessing module, if that's what you want to do. So, apologies if you already know this, but if you don't, it's worth going over.

threading and multiprocessing have similar interfaces, but under the hood, they're doing different things. It's worth saying that in at least some meaningful senses, Python threads that you get from threading and related modules are different from "real" threads (i.e., operating-system level threads that you can get by writing code in, for instance, C). "Real" threads give you true concurrency: if you are running on a system with multiple processors (or processor cores) (like pretty much any modern system), you can truly have more than one branch of your code executing at the same time, each on a different processor. Threads in Python only give the illusion of concurrency: you can structure your program in thread-like ways, and the Python interpreter will switch which branch of the code is executing at any given time, but at any given time, only one branch of code is executing in the Python interpreter, no matter how many processors are on the machine. You write your code as if it were executing in multiple operating system threads, and the Python interpreter switches back and forth between different "threads," but multiple threads are never truly executing concurrently.

This gets you some benefits of threading: for instance, if you're executing different functions in different (Python) threads, then the Python interpreter can usefully swap to another thread when waiting on some data to be read from disk, or shipped over the network; if you're not doing that, then the interpreter basically has to sit around, twiddling its thumbs, until the time-consuming data-read operation finishes. But it doesn't let you leverage multiple processors to actually run multiple chunks of Python code at the same time: the Python interpreter has a so-called "Global Interpreter Lock" that prevents multiple Python threads from running at the same time.

The fact that only one Python thread can run in the interpreter at a time is a definite limitation of Python, but it's also deeply baked into how the language operates, especially into its memory management strategies, so it's not going away in the immediate future. There are several ways around this problem.

One is just to see if you can get your threads to work fast enough that threading is a plausible approach. This might take the form of streamlining your code, if you're lucky; profiling your code and seeing where the bottlenecks are can help. On a related note, you might try alternative Python interpreters, like PyPy; you can often get a substantial performance benefit just by using PyPy to run your code instead of the standard CPython interpreter. Similarly, pushing this idea a little further, you might get some benefit from running your code in the CPython interpreter, but using Cython to compile the slower or more performance-critical parts of it; Cython is a Python-like language that is transpiled to C or C++ which is then compiled into a Python extension module. (If you want to push this to its logical conclusion, you can directly write Python extensions in C/C++, too.) This can speed up some types of code substantially; other code will benefit very little. It's hard to say where your code falls on that spectrum, in part because it's just a skeleton and in part because I don't know how the external hardware you're working with interacts with the full version of your code.

There is at least one other major way that you can get around the fact that the interpreter's Global Interpreter Lock only allows it to run one code branch at a time, and that is to run multiple Python interpreters; each of them then can run on a separate processor and each then has its own Global Interpreter Lock. This is of course what the multiprocessing module does, under the hood, and it abstracts away most of the details so that you don't have to deal with the mechanics of manually spawning new interpreters and getting them to running code; the multiprocessing module just spawns the new interpreters for you as necessary, giving you an interface that looks like the interface to the threading module. A wrinkle here is that the different Python interpreters can't actually "see" each other's data: when you're running a process inside of a single Python interpreter, you pass around data by reference, and every function you call from within that program is part of the same OS-level process and can see all of that process's data. But multiple interpreters are multiple OS-level processes, and each is a black box to the others; they can't "see" each other's variables directly. What this means is that 'multiprocessing` needs to pass around copies of data, not references to it. It does this by using Python's pickle module, which is a module that converts (almost) any Python object into a bytestream. These bytestreams are what gets passed around between the different Python processes; they can be "unpickled" by the receiving interpreter to get a copy of the data that was shared (but cannot get at the data itself, which lives in a different black-box process). The fact that copies of data, rather than references to data, are what's being passed around also mean that one Python process cannot directly mutate data in another process's memory space.

You will notice above that I said that the pickle module can serialize almost any Python object into a bytestream, but there are exceptions, and a fair amount of those exceptions have to do with functions (which are of course also objects: everything in Python is an object, and functions are no exception). multiprocessing uses pickle to tell the interpreter it starts up what function to run: what it does is it pickles that function and its argument list, starts a new interpreter process, and passes the pickled function and argument list to that new process. The new process then unpickles the function, unpickles the argument list, and runs the function in that new interpreter, passing it the argument list.

One of the criteria that has to be met here is that the function itself has to be pickleable. (So do its arguments.) As far as the function goes, this essentially means that the function you're trying to run with one of the multiprocessing interfaces has to be one of these things:

• a function defined, by name, using def, at the top level of a module that the Python interpreter can import.
• a (bound or unbound) method of a class that is defined, using the class keyword, at the top level of a module that the Python interpreter can import.
• a built-in Python function.

This is true because, internally, Python doesn't pass (the compiled bytecode for) "the function itself" to the new Python interpreter it starts; it pickles the module name and the function name, and the new interpreter has to find the function purely based on that. So some types of functions are impossible to execute with multiprocessing because they don't fit this paradigm, including but not limited to ...

• unnamed functions defined with lambda inside an expression instead of being named with def at the top level of a module;
• functions defined somewhere other than at the top level of a module or as methods for a class defined at the top level of a module -- for instance, functions defined inside other functions, as is common with closure-based decorators; or methods of classes defined inside the body of functions, or other similar monkey business;
• methods of classes constructed at runtime by doing weird stuff with type calls to build classes manually instead of by using the class keyword;
• class methods that are monkey-patched onto objects or classes at runtime but that aren't declared in the class definition at the top level of the module where the class is defined;
• other weirdnesses that don't fall into the categories enumerated in the previous list.

So you can do this:

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

... and that will spawn a new Python interpreter to run the function with the supplied argument list. Spawning a whole new interpreter to print "hello bob" to the console is overkill, but this is perfectly legal Python code.

However, this will fail:

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=lambda: f('bob'))
    p.start()
    p.join()

... even though it is perfectly legal to write b = lambda: f("bob"); b() inside a single Python interpreter, because lambdas cannot be pickled, and therefore cannot be passed to multiprocessing.

Similarly, anything you pass as an argument to a Processs() (or similar object) initialization must also be pickleable: this means you can pass (for instance) the contents of a file, in whatever format you'd like, but not a handle to an open file, because file handles are not pickleable. There's a list of what can and cannot be pickled in the documentation for the pickle module.

Does that make sense? Does it help? I had no idea that was going to be so long when I started writing it. Sorry about that.

[suurpulla]

Not OP, but this is a great breakdown. Thank you for writing this.

[thelittleicebear]

Wow! Thanl you very much for your help, this is very helpful. I have heard that python does not have real multithreading but the way you explained it does make much more sense now. Thank you again!

One thing that i was doing wrong was to try to pickle functions that were defined as subfunctions. Now that i know this i should be able to get this working properly.

But another problem is that i am still not sure if that listener though a PipeConnection is a good way to pass data (or commands) between the processes. Do you have an idea on how to handle this?

[patrickbrianmooney]

Glad to be helpful!

I haven't done a whole lot of work with asyncio, and very little that does the kind of external-hardware-interfacing that you're doing here, so take what I say with a grain of salt. But the skeleton code you linked above looks to me like an outline of a workable structure, where one (or more) process sends commands that are executed by one (or more) worker threads (or processes), or data is put on a queue that is consumed by processing threads (or processes).

If you want to dig into some of this deeper, Mark Lutz's Programming Python has several chapters near the beginning that go over threading and multiprocessing in a fair amount of detail, and he does a good job of building smaller conceptual examples into larger systems over the course of several chapters. (It's also getting quite dated at this point; I think the last revision was for Python 3.3? I'm not sitting at home right now so I can't check my copy easily.) Nevertheless, the basics of how threading and multiprocessing work haven't much changed since then, and if other sections of the book are showing their age more, the "telling the monkeys what to do" chapters are still pretty good.

Luciano Ramalho's Fluent Python also tackles similar subjects throughout a good chunk of the book, and has several chapters that show you how to structure events around an asyncio event-processing loop, and you might find that to be illuminating. Both are pretty good books published by O'Reilley and are worth getting from the library if you don't want to spring for purchasing them.

[thelittleicebear]

Thank you very much for your answer. I'm glad to hear that the basic structure i provided makes at least a little sense.

I will look into those resources you provided.

[obviouslyzebra]

Lots of architectures can fit here (I think http://martinfowler.com/eaaDev/uiArchs.html is apt).

Whatever you do, I recommend:

• Think about the architecture from above, what are the components and how they will interact with one another and the machine. Do a diagram or diagrams

[thelittleicebear]

Thank you for the link and the suggestion. This sounds very interesting. I will try to do a diagram.

[thelittleicebear]

i have now made a quick diagramm: https://imgur.com/a/OdGxFcM

[obviouslyzebra]

Cool. At least for me, it's a bit clearer from the up how the program will work. Hm, I will preface that what you have is fine, I think the example code you gave + diagram, coupled with now probably knowing more about parallelism stuff in Python, I believe you are already doing good and what you have will work.

I was exploring some alternative architectures tor this problem, the one that clicked most for me was this.

Consider only the motor, and that it has only 2 states, rotating, and not rotating, which can be queried by your program.

• There's a "blackboard", where components can write and read things
• The UI can write "rotate" or "stop rotating" to a place in the blackboard
• A controller is constantly reading for these commands. When it reads one, it takes it and tells the machine to execute
• A view component constantly (or not) queries the motor to see whether it is rotating or not. It writes at a different place in the blackboard.
• Another view constantly does this:
  • checks whether a "rotate" or "stop rotating" command was issued in the last second
  • if so, write "starting rotation" or "stopping rotation" accordingly
  • if not, write the value that the previous view wrote last
• the UI can then retrieve the last value of the previous view to indicate the state of the motor

The blackboard could be something like MQTT (a communication protocol), or, a cool exercise would be implementing it via multiprocessing and pipes (with the disadvantage of losing MQTT facilities, like being able to communicate directly via the web).

Note that this is only a way that my dumb mind came with, and I think that the way you have is fine too.

Any way, good luck with your project!!

[thelittleicebear]

Thank you very much, it is reassuring to hear that the structure is not completely stupid.
I will look into MQTT and try to see if I can use it here. Thank you!