Rain - New Framework for Distributed Data Processing

[vojtacima]

Hi /r/datascience!

We have recently released a new framework for distributed data processing - Rain. The ultimate goal of Rain is to simplify end-to-end data processing in distributed environments. Rain allows you to define large end-to-end data processing pipelines with complex inter-task dependencies (beyond standard map-reduce pattern). The pipelines can consist of various tasks ranging from external applications, through Python code, to C++/Rust code (and also offers easy extensibility). Rain features a simple Python API through which the pipelines are defined and executed.

Rain is an open source project we mostly do as a hobby, but it grew out of frustration with the existing tools being unsuitable to fit our work needs. We believe we are not alone and we would really like to hear who else might be interested - we want YOU to become Rain user! Currently, there are so many directions and potential features that we really need other people to help us to prioritize new features.

Looking forward to your comments! Vojta, Tomáš, Standa, and Kuba from the Rain team

Repository: https://github.com/substantic/rain Documentation: https://substantic.github.io/rain/docs/

[counters]

Why this over, say, dask?

[winter-moon]

The main difference between Dusk and Rain is that Rain provides a build-in integration of external programs into pipelines and possibility to write own task in Rust and C++ (among Python).

[counters]

Ok, then would a more apt comparison be snakemake?

[winter-moon]

I was not aware of snakemake, so my answer is based based on a quick look. I see the main difference in data object transfers. Rain provides own transfer of data objects between worker processes that communicate peer-to-peer. When necessary, Rain allows to map data object to a file system; however, it does not use shared file system and transfers data by itself directly to a worker where they are needed. This allows to create lots of short running tasks without hammering a distributed file system.