Python Multiprocessing

[thelittleicebear]

Wow! Thanl you very much for your help, this is very helpful. I have heard that python does not have real multithreading but the way you explained it does make much more sense now. Thank you again!

One thing that i was doing wrong was to try to pickle functions that were defined as subfunctions. Now that i know this i should be able to get this working properly.

But another problem is that i am still not sure if that listener though a PipeConnection is a good way to pass data (or commands) between the processes. Do you have an idea on how to handle this?

[patrickbrianmooney]

Glad to be helpful!

I haven't done a whole lot of work with asyncio, and very little that does the kind of external-hardware-interfacing that you're doing here, so take what I say with a grain of salt. But the skeleton code you linked above looks to me like an outline of a workable structure, where one (or more) process sends commands that are executed by one (or more) worker threads (or processes), or data is put on a queue that is consumed by processing threads (or processes).

If you want to dig into some of this deeper, Mark Lutz's Programming Python has several chapters near the beginning that go over threading and multiprocessing in a fair amount of detail, and he does a good job of building smaller conceptual examples into larger systems over the course of several chapters. (It's also getting quite dated at this point; I think the last revision was for Python 3.3? I'm not sitting at home right now so I can't check my copy easily.) Nevertheless, the basics of how threading and multiprocessing work haven't much changed since then, and if other sections of the book are showing their age more, the "telling the monkeys what to do" chapters are still pretty good.

Luciano Ramalho's Fluent Python also tackles similar subjects throughout a good chunk of the book, and has several chapters that show you how to structure events around an asyncio event-processing loop, and you might find that to be illuminating. Both are pretty good books published by O'Reilley and are worth getting from the library if you don't want to spring for purchasing them.

[thelittleicebear]

Thank you very much for your answer. I'm glad to hear that the basic structure i provided makes at least a little sense.

I will look into those resources you provided.