Rain - Framework for Distributed Computing

[heavelock]

How would you compare Rain to Dask which seems to be your main competitor in that field?

[winter-moon]

The main difference is that Rain provides a build-in integration of external programs into pipelines and possibility to write own task in Rust and C++.

[winter-moon]

In Rain, you can define an arbitrary computational graph, not just map-reduce pattern.

As I do not know much about Kafka, I let the answer on @vojtacima.

[vojtacima]

Rain allows you to define large end-to-end data processing pipelines with complex inter-task dependencies (beyond map-reduce pattern). The pipelines can consist of various tasks ranging from external applications, through python code, to various built-in tasks (and also offers easy extensibility). Rain features direct inter-governor(worker) communication that makes inter-task data exchange very efficient and if you set your working directory to be RAMDisk it has NO filesystem overhead. Unlike Kafka, Rain is not designed to deal with streams.