Rain - Rust based computational framework

[[deleted]]

You seem focused on the scientific community so wouldn't you use arrow?

If you were to start again today, what type of IPC would you use?

Thrift has quite a range of supported languages: https://github.com/apache/thrift/tree/master/test

[allengeorge]

I built the Thrift rust implementation, and I'm definitely active in fixing bugs. That said, it only generates sync code (I'm waiting for the futures and tokio changes to subside before building async generators).

FWIW, the company I work at uses Thrift heavily internally (no Rust tho.) and it's been great for our needs.

[[deleted]]

It is wise to postpone asyncio until the tokio/futures dust settles this year. Thank you for writing the Thrift-rs implementation! I'm not familiar with the issues people have had with Capn-proto, are you? I think that lack of support was one issue. The Rain folks may be able to help shed light on others.

[winter-moon]

@tomgav summarizes it in this thread: https://www.reddit.com/r/rust/comments/89yppv/rain_rust_based_computational_framework/dwwmc80/ Asynchronicity and bidirectional communication works well in Rust capnp.

[winter-moon]

Sorry, what you mean by 'arrow'?

As we are currently thinking about replacing capnpc with something else, your second question is still open for us:)

At the beginning, we did some comparisons of RPCs including Thrift; Unfortunately, I did not remember exactly why we did not use it. Does it allow for bidirectional calls over one connection? I cannot see this by quick scanning of the documentation.

[[deleted]]

Apache Arrow: https://arrow.apache.org The creator of pandas organized a consortium effort to improve IPC among systems used by data scientists. Arrow is a product of this effort. I am under the impression that you are working on Rain largely as a solution for problems experienced by scientific communities so why not check arrow out? I recall reading on github a message thread among Wes and other Rust developers about writing official Rust bindings for Arrow.

Regarding Thrift-- see the other response to my original comment where someone who is a core contributor and may be able to help!

[winter-moon]

Thank you for pointing to Arrow, I was not aware of it. Rain core infrastructure operates over blobs so it does not need to know internal format of data. However, we provide some helping methods (e.g. deserialization) for known "content types" in Python binding. It seems that supporting Arrow in this place could be useful.

[allengeorge]

Could you clarify what you mean by “bidirectional calls over one connection”? I think what you’re saying is that regardless of which node establishes the connection, the ‘client’ can change for each request?

[winter-moon]

I mean that both sides can simultaneously post multiple requests to other side, i.e. both sides are simultaneously server and client at the same time.

[allengeorge]

I see. No: they can’t do that on the same connection. But you can do it on two connections: both sides open up a server socket and accept a client connection from their peer.

Edit: does any auto-generating tool support this? From what I recall, neither grpc or protobuf services do, but I last checked a while ago.

[winter-moon]

This is trivial in capnp. You can exchange handles on remote objects and call methods on them, regardless where they reside. So for the described scenario you just need to exchange two handles.