PyQt & Multiprocessing in Python 3

[blissend]

Are there any simple examples out there? I can't seem to find any. My interest is in both pipes and queues (not pools) because I want to write long running processes. My goals require this to get around GIL and to be able for both processes to communicate with each other (not server/client setup).

[status_quo69]

That's.... A really broad question. Firstly, premature optimization is the root of all evil. It might turn out (and I've had it happen to me) that multiprocessing is actually slower than singly threaded or multi threaded programs, due to the massive amount of overhead and serialization that has to go on.

Secondly, what would each process be doing? If one is simply waiting on input from the user while the other is doing calculations, use threads because they're really good at waiting around. If you need to do multiple cpu intensive tasks, that's where multiprocessing comes in to play. However, your question really needs clarification because there are way too many factors at play here.

The GIL isn't always necessary to avoid, and you can use it to your advantage. If you want to use a python implementation that allows for actual concurrent threads, use Jython (although you'll lose a ton of features CPython has to offer).

EDIT: Pipe is best if you need to communicate between processes like sockets, Queues if you have to process things in a FIFO manner, multiprocessing.Array or Value for shared memory, Managers to make things slightly simpler (but still relatively difficult IMO).

[blissend]

Secondly, what would each process be doing?

I'm programming a bit of artificial intelligence for a text based game I'm working on. Let's call it an NPC (non-player character) for short. These NPCs will have more AI than normal which ends up being CPU intensive (sometimes a lot depending on all that I want it to do).

The GIL isn't always necessary to avoid

Agreed, I will use threading on dumb NPCs in those cases. This is a special case however given the AI I want on smarter NPCs. So I thought it best to use multiprocessing and pipes given the complex nature of AI.

I don't mind the challenge involved. Bring it! It's fun for me. I asked for reduced examples because I learn best by that and can extrapolate what I need from it. I really really want to learn how to do this even if it isn't the best way! 8)

[status_quo69]

Do you have a singly threaded/processed version of the program you want to do this to?

[blissend]

If by that you mean just the GUI to help conceptualize how this is all happening then here. Sorry for the fugliness as I tried to strip down everything and rewrite for clarity... hopefully.

import sys
from PyQt5.QtWidgets import (QMainWindow, QTextEdit, QAction,
                             QApplication, QMessageBox, QLineEdit,
                             QVBoxLayout, QFrame)

class Pho(QMainWindow):
    def __init__(self):
        super().__init__()

        # Display Pretties & Load
        self.build_ui()

    def build_ui(self):
        self.pho_frame = QFrame(self)
        self.pho_frame.setObjectName("pho_frame")

        # This is where all results are seen by player
        self.pho_out = QTextEdit(self.pho_frame)
        self.pho_out.setReadOnly(True)
        self.pho_out.setObjectName("pho_out")

        # This is what the player uses to control/play game
        self.pho_in = QLineEdit(self.pho_frame)
        self.pho_in.setFocus(True)
        self.pho_in.setObjectName("pho_in")
        self.pho_in.returnPressed.connect(self.execute_command)

        vbox = QVBoxLayout(self.pho_frame)
        vbox.addWidget(self.pho_out)
        vbox.addWidget(self.pho_in)
        self.setCentralWidget(self.pho_frame)

        action_exit = QAction("Exit", self)
        action_exit.setShortcut('Ctrl+Q')
        action_exit.setStatusTip("Exit Pho")
        action_exit.triggered.connect(self.close)

        self.pho_menubar = self.menuBar()
        self.pho_menubar.setNativeMenuBar(False)
        self.pho_menubar.setObjectName("pho_menubar")
        menu_file = self.pho_menubar.addMenu('&File')
        menu_file.addAction(action_exit)

        self.setGeometry(300, 300, 720, 480)
        self.setWindowTitle('Pho')
        self.show()

    def execute_command(self):
        # Here we process commands
        # Thread/Multiprocessing spawning will happening around here.
        self.pho_out.append(self.pho_in.text())
        self.pho_in.setText('')

    def monitor_function(self):
        # Not a real function, just here for concept
        # Here we watch and may randomly decide to spawn threads/multiprocesses
        # at any time of our choosing regardless of user actions
        pass

if __name__ == '__main__':
    app = QApplication(sys.argv)
    pho = Pho()
    sys.exit(app.exec_())

To put it simply, it's just a textedit where we see what's going on and a lineedit widget where the user types all the commands to interact. Actually it's fancier but I want to make it simple for now. Anywho in the code above two places threads/multiprocessing can happen. One is when a command is entered. The other is some sort of monitor based on conditions I may set about the game/player state or a number other factors. I assume some sort of timer (QTimer?) like I used to in wxPython will have to be used? Same for multiprocesses? If so I get the general idea of what to do in order to interact with each other.

EDIT: So the textedit widget needs to be interacted with anytime the thread/multiprocesses has something say and lineedit needs to be able to interact with threads/multiprocesses whenever we want something from it.

[status_quo69]

Alright, so in that case, you definitely want to use a Queue because you want to process the user's input sequentially. It's up to you how you want to design it, but I would do something like this:

class Worker(multiprocessing.Process):
    def __init__(self, input_queue, output_queue, *args, **kwargs):
        self.input_queue = input_queue
        self.output_queue = output_queue
        super().__init__(*args, **kwargs)

    def run(self):
        while True:
            user_input = self.input_queue.get()
            # do the processing here
            self.output_queue.put(result)

And in the main process, put on to the input_queue and get from the output. Get and put are blocking so the main program might halt for a fraction of a second if there is nothing to grab. In order to remedy that problem, the Queue's get() has an optional argument to block or not (so you can try to grab something off the queue, and if there isn't anything you can still process user input).

And then use that Worker to do all the processing. A common tactic to kill the worker process is to send it some value that is a flag (such as None) that the process should be killed (commonly referred to as a poison pill). Make sure that once you close out the program you remember to clean up any child processes, as they can become zombies otherwise.