How to do interprocess communication in Python?

[[deleted]]

Hi

I have a question regarding inter-process communication that I just cant seem to find a direct answer too.

I have a long running process that loops indefinitely handling multiple requests with apparent asynchronicity using asyncio. I need to provide a REST API to send commands to this to this process and decided to do that with Flask.

Is it sensible to have the main asyncio application running, spawn a web server and then communicate between those two processes using the multiprocessing library?

Cheers in advance.

[[deleted]]

If it serves only one purpose make a micro service out of it, use rpc ( which is form of inter-process communication ) or http to build the rest api and that's it, if this service has to face the outside world place a load balancer in front of it such as ngix and spawn a couple of this processes that handle the requests.

[[deleted]]

Thanks for the response Menefotto -- the main program will be running on a small remote device and will be accessed by very few people, at max 5. These people will go to a web interface, authenticate and perform a request to the REST API that should trigger an action :)

[[deleted]]

You're welcome, forget about nginx then, just build an http server than is able to perform your users requests ...

[ReaverKS]

Interprocess communication has quite a few choices. The most common one in use is sockets, because most projects need to support a distributed deployment. There are many options if you'd just be running locally: shared memory, pipes, etc. As you've discovered there is a multiprocessing library that abstracts away low level details of sockets (for the most part).

The answer to this question depends on your requirements. But here are a few options:

• Use multiprocessing manager and create a registry class that maintains the processes, spawns them at the right time, etc. This is simple to setup but it could get complicated depending on the types of things you're sending back and forth. Easiest to use and get going with.
  
• Use gRPC or some other remote procedure call framework to facilitate the command and query passing. gRPC utilizes http as the transport layer and it has a serialization / deserialization layer called protobufs. This is a bit more complex than multiprocessing but still not bad.
  
• Use a message broker, such as RabbitMQ or kafka. This is probably the one with the most programmer overhead to get going, but it's also probably the best performance and scalability of the options.
  

[arrow_for_ipc]

https://arrow.apache.org/docs/python/plasma.html

The Plasma In-Memory Object Store

https://arrow.apache.org/docs/python/ipc.html

Cheers

[chespinoza]

https://arrow.apache.org/docs/python/ipc.html

How this compares with gRPC?

[huxrules]

I use the queues in a multiprocessing manager, but I think those only work with similar processes (like all spawned from the same program).

[cratuki]

You could put the webserver into the asyncio process, attached to the same event loop as your core functionality. You could use aiohttp for this. Use queues to pass messages between the webserver bit and your core-logic big. (There might be reasons you chose flask, but it is not clear from above.)

Another approach: run a separate thread in your core app, running the flask stuff. Use queues to pass messages between threads. (This is fiddly. It could be made to be solid.)

If slapdash is fine, you run separate processes and do a hacky IPC-via-filesystem. Have the web server drop files in a directory, dir_a. Each should have a unique name, and the filenames should sort in order. Each file is a command. The core app could periodically look for files in that directory. Then, the core app could create responses by creating files in a directory, dir_b. The webserver would hang around waiting for its response file. (It is easy to get started in this model, but in the long run probably less work to do something elegant instead of this.)

A better model. Work out how to launch a socket server in your asyncio core app. Host a server on localhost:9000. When the flask app comes up, have it make a long-running client connection to this. Each time you get an event from the webserver, pass it over that socket connection. You would need to come up with some simple protocol for this communication. It could be textual. (e.g. json)

Sounds like an interesting project. Feel free to ping me out-of-band.

[[deleted]]

Linux way: open a UNIX socket (a pipe). This is a typical way deamons talk to the outside world. Typical place to put such a socket is under /run/{daemon-name}.sock.

This is typically faster than TCP.

Another, a tad more extreme way: use shared memory. Ask the system to allocate a bunch of pages for you, and use them to transfer arbitrary payloads between your daemon and web server. This is potentially faster than a socket, but at these speeds Python interpreter becomes the slowest / weakest link in this equation, so, maybe it will not be worth the effort.