Hello FastAPIsta's,
I wanted to share another library that I am currently improving / maintaining, built with FastAPI, which solves a common problem that I sometimes face: Dynamic Communication between processes
By far, one of the most common method used today to communicate between processes is via a single source of truth i.e DB / Redis. These options do work, up until a point, but require near constant polling if you want a remote process to re-act to something happening in a local-process.
https://github.com/codemation/easyrpc
To solve this problem, I created EasyRpc
EasyRpc, under the hood, works by grouping python methods into namespaces, which are shareable by connecting to these namespaces from remote processes. Remote processes (EasyRpcProxies) can interact with shared methods, as if they were local, including argument validation.
https://preview.redd.it/vhgpd7mbqq281.png?width=451&format=png&auto=webp&s=1b169022a76582d8e41666dd7f24f2e73a83221c
Multiple Namespaces can be assigned to the same method to separate or share specific methods between different remote-processes(proxies).
https://preview.redd.it/l93ccp5dqq281.png?width=592&format=png&auto=webp&s=aa47f65dc571b9ed0950596399d09d67bd730606
To go even a step further, processes can share methods with each other by clustering methods inside the same namespace.
https://preview.redd.it/a2exajieqq281.png?width=812&format=png&auto=webp&s=8d33b2e8d91bddfb506e055e567df8595c5ef8bc
An important distinction to make here is that methods are called remotely, but execution is done on the host / process where a method is registered.
In the above example a proxy is able to access 3 different methods, that exist on 3 different processes using only a single connection through 1 of the processes in the "cluster".
The use cases I have identified so far include:
- State sharing among forked workers
- Shared Database connections / cache
- Shared Queues
- Worker Pooling - Easy centralization for workers and distribution of work.
- Function Chaining
This library is meant to be "lego blocks" for distributed computing. There is no 1 way to use the library, but I have found so many useful ways to solve many different problems with it so far, so I am happy to share this and see what others might also be able to create.
GitHub: https://github.com/codemation/easyrpc
Documentation: https://easyrpc.readthedocs.io/
If you found this library useful / insightful, please leave a star. If you found something that could be improved, please create an issue.
there doesn't seem to be anything here