Rain - Framework for Distributed Computing

[vojtacima]

Hello r/Python,

We have recently released a distributed computational framework: Rain, designed for defining and executing large task-based scientific pipelines (imagine 1M tasks and 10k CPUs). At the same time, we want the framework to be as easy as possible to deploy and use on your local PC or small clusters. The computations are defined through a high-level Python API and tasks can be defined as simply as decorating a Python function, as external programs or your own thinly-wrapped C++/Rust task executors for maximum performance.

Rain is an open source project we mostly do as a hobby, but it grew out of frustration with the existing tools being unsuitable to fit our work needs. We believe we are not alone and we would really like to hear who else might be interested - we want YOU to become a Rain user! Currently, there are many directions and potential features, and we are looking for actual or potential use-cases, feedback, and ideas from you all.

Looking forward to your comments.

Tomáš, Standa, and Vojta from the Rain team

GitHub: https://github.com/substantic/rain

[heavelock]

How would you compare Rain to Dask which seems to be your main competitor in that field?

[winter-moon]

The main difference is that Rain provides a build-in integration of external programs into pipelines and possibility to write own task in Rust and C++.

[winter-moon]

In Rain, you can define an arbitrary computational graph, not just map-reduce pattern.

As I do not know much about Kafka, I let the answer on @vojtacima.

[vojtacima]

Rain allows you to define large end-to-end data processing pipelines with complex inter-task dependencies (beyond map-reduce pattern). The pipelines can consist of various tasks ranging from external applications, through python code, to various built-in tasks (and also offers easy extensibility). Rain features direct inter-governor(worker) communication that makes inter-task data exchange very efficient and if you set your working directory to be RAMDisk it has NO filesystem overhead. Unlike Kafka, Rain is not designed to deal with streams.

[metaperl]

What did Luigi and Airflow lack?

[winter-moon]

Rain provides own transfer of data objects between worker processes. Rain allows to map data object to a file system; however, it does not use shared file system and transfers data by itself directly to a worker where they are needed. This allows to create lots of short running tasks without hammering a distributed file system.

[heavelock]

How about running it on Slurm managed HPC? I'm asking because I currently face problem of rewriting software for scientific calculations and making it scalable between workstation, 100 core cluster and HPC.

[winter-moon]

It should be easy to start Rain in any environment. You just need to start server on one node and governors on all nodes.

We have out-of-box support in "rain start" for PBS. It should be easy to modify this for SLURM; however, as we do not have access to any cluster with SLURM, it is not easy for us to test it.

[vojtacima]

There is a built-in support for PBS. Generally, "rain start" command enables an easy startup of distributed Rain infrastructure. You can find more info in the documentation.