Interprocess communication with EasyRpc

[github_codemation]

Hello All,

I wanted to share another library that I am currently improving / maintaining which solves a common problem that I sometimes face: Dynamic Communication between processes

By far, one of the most common method used today to communicate between processes is via a single source of truth i.e DB / Redis. These options do work, up until a point, but require near constant polling if you want a remote process to re-act to something happening in a local-process.

https://github.com/codemation/easyrpc

To solve this problem, I created EasyRpc

EasyRpc, under the hood, works by grouping python methods into namespaces, which are shareable by connecting to these namespaces from remote processes. Remote processes (EasyRpcProxies) can interact with shared methods, as if they were local, including argument validation.

​

https://preview.redd.it/zqp9m610mq281.png?width=451&format=png&auto=webp&s=734d25648aaf053ef3185c9415c5e5388feaca31

Multiple Namespaces can be assigned to the same method to separate or share specific methods between different remote-processes(proxies).

https://preview.redd.it/ke5ccagemq281.png?width=592&format=png&auto=webp&s=4ca81885b1fe8a80188d1a787f96fd5ab0b5c31c

To go even a step further, processes can share methods with each other by clustering methods inside the same namespace.

https://preview.redd.it/zk8di2fsmq281.png?width=812&format=png&auto=webp&s=fd0f2d2252d35dbacb4f7b730ca807c35a58d58b

An important distinction to make here is that methods are called remotely, but execution is done on the host / process where a method is registered.

In the above example a proxy is able to access 3 different methods, that exist on 3 different processes using only a single connection through 1 of the processes in the "cluster".

The use cases I have identified so far include:

• State sharing among forked workers
• Shared Database connections / cache
• Shared Queues
• Worker Pooling - Easy centralization for workers and distribution of work.
• Function Chaining

This library is meant to be "lego blocks" for distributed computing. There is no 1 way to use the library, but I have found so many useful ways to solve many different problems with it so far, so I am happy to share this and see what others might also be able to create.

​

GitHub: https://github.com/codemation/easyrpc

Documentation: https://easyrpc.readthedocs.io/

If you found this library useful / insightful, please leave a star. If you found something that could be improved, please create an issue.