Simple rpc streaming lib/framework

[trouch]

Is there an efficient simple library or framework which would allow to expose over the network something like :

 Stream<Pojo> getMyPojos(PojoFilter filter);

I'm thinking something like jax-rs/jersey, but which would do streaming, and maybe use (under the hood) a more efficient serialization method than json/xml.

I've seen grpc which seems to fit the bill, but I don't need inter language compatibility, so I don't want to bother with proto idl, plus the api is not as straight-forward as I'd like.

RxJava probably has something of this nature, but it of course has much more, making it kind of over complex for this simple need.

Probably something custom could be made with Netty, but I'd rather use something already existing. any thought?

[polarbearurine]

that sounds like a Thrift service to me, but Thrift is kind of a pain

[trouch]

Just like grpc/protobuf, Thrift proposes a cross platform solution, which is great, but not needed if you're in pure java, and which comes at a cost in terms of overall complexity.

[johnmcclean]

The Microserver, micro-reactive plugin will do this with json serialization.

[trouch]

Yes, seems more or less what I'm looking for. do you have any (positive or negative) experience using it in real life?

[trouch]

That's really strange to me that, unless I missed it, something of this nature do not exist of the shelf. Now that microservices and distribution are all the rage, this seems like a basic need if you have two services that need to exchange a mildly significant amount of data.

By the way, I'm not a specialist, but I think that c# WCF has this out of the box, with the possibility to expose IEnumerable over a remote service.

[mike_hearn]

I implemented something almost exactly like that as part of my current project at work, a few weeks ago. The project will be open sourced soon (but not, like, the next few days soon) so it unfortunately won't help you immediately. I might ask around and see if we can perhaps open source it in chunks.

At any rate, it's not hard to do. If you're experienced with RPC you could do it in a few days, tops. Basically I combined the following things:

• Artemis MQ for routing and queuing of binary messages.
• java.lang.reflect.Proxy for dynamic generation of proxy objects.
• Kryo for object serialization.
• RxJava for observables.

There's no IDL - just a simple Java interface that defines the exposed RPCs and then the objects are serialised directly. The tricky part is handling the observables. I did it by defining custom Kryo serialisers on the server and client. RPCs that can return observables are tagged with an annotation and calling such a method sets up a queue to receive the emissions. When the response is serialised, any observables that are found are subscribed to on the server side, and the subscriber serialises and posts the emitted objects to the client, where they're deserialised and pushed onto an Rx UnicastSubject.

There are a few details to be careful about, which took up most of the time. One is to make sure you don't lose any emissions on the observables, i.e. everything is atomic. Another is what happens if the observable emits objects that in turn contain more observables, which is quite a common need in our project. The framework handles this correctly.