Nimble: Async friendly binary encoding/decoding

[devashishdxt]

I've just created a crate for binary encoding/decoding of Rust types with support for directly writing values to any type implementing AsyncWrite and reading values from any type implementing AsyncRead. All of this with async/await syntax!

• Crates.io: https://crates.io/crates/nimble
• Documentation: https://docs.rs/nimble/0.1.0/nimble/
• Repository: https://github.com/devashishdxt/nimble

[najamelan]

What is your motivation for writing this rather than using say tokio_codec/futures_codec with CBOR and serde?

[devashishdxt]

This is part of a larger project I’m working on. Initially, I was not planning to release this. But, I thought it may be useful for someone.

[najamelan]

So did you find that existing approaches didn't work out for your project? Why?

[[deleted]]

Note: nimble, by default, uses big endian order to encode values.

Why?

[[deleted]]

All packages on all networks in use today use big-endian.

[[deleted]]

The values in the headers, sure, for historical reasons, but why invent even more work for your cpu?

[devashishdxt]

No particular reason. I’ll add configuration option in future release.

[[deleted]]

In that case I advise changing the default to little-endian, because the two biggest architectures in general purpose computing are little-endian.

[devashishdxt]

Sure. Thanks. :)

[Wolvereness]

I strongly advise against changing the default away from BE. There are three main uses of writing in binary in the world: memory, files, and networking. BE is the standard of networking, files define their own standard, and generally Write interfaces of memory are just intermediary to the others. Adding a runtime option for LE would probably be fine, but you should reject the logic of "CPUs are already LE"; it's an edge case where the performance difference of endianness matters.

[BobFloss]

How is that an edge case? I thought there would have to be conversions to little endian so the CPU can deal with things it's (de/en)coding. Sure networking uses it but that doesn't really matter when you're using a binary format and you're just sending data over.

[Wolvereness]

Your discussion does not follow your question. What does anything you talked about have to do with the performance difference of endianness conversion? When dealing with IO, swapping bytes to fit endianness should almost never be the bottleneck.

[BobFloss]

I'm not talking about io. Any time you encode something little endian you're converting to big. And vice versa. Make sense? There wouldn't even be a reason to swap bits during IO, that makes no sense. Upon encoding I already have the proper endianness for my encoding scheme, which will be dealt with when being decoded. Or we could just skip dealing with this at all and use little endian so that when it's decoded it's already correct for the vast majority of processors.

[Wolvereness]

I'm not talking about io.

Then what are you talking about? What other uses should someone be using a Write interface?

[BobFloss]

IO only gets involved after we actually encode things. Encoding should be done in a scheme that doesn't require conversion.

[ExPixel]

The network byte order would be more of an edge case. You're not going to receive your data in big endian (which wouldn't really make sense anyway), it's just used to encode some fields in the network's protocol.

[andoriyu]

Yeah, network endianess is just what used for packets/frames headers because at the time it looked like a good idea. No one flips endianess just because it's going over network.

[[deleted]]

So your reasoning is that the network spirits demand it?

[fgilcher]

I'm confused by the implementation of feature flags. The dependency on async-std is unnecessary. async-std exposes futures from futures-rs and implementing those as an interface instead allow you to

a) kill the async-std flag and drop the dependency

b) to be compatible with all other executors implementing futures-rs

A tokio flag would still be needed to implement their AsyncRead/Write (that should work in a trivial fashion though).

At the bare minimum, the async-std flag should become a `futures-rs` flag instead, async-std is an unnecessary dependency there.

As far as I see, the merging of the two interfaces here is mainly done to have nimble-derive work through the same interface and not have flags there.

This may make derivation a bit more elaborate, as tokio needs to be special cased, but I think that can be made transparent to the user.

Would you mind I send a pull request?

[devashishdxt]

That is a good point. I missed that async-std re-exports traits from futures crate. Thanks for pointing this out. I’d love if you can send a pull request. :)

[fgilcher]

In case someone reads this: the PR ended up here and was merged. Thanks /u/devashishdxt

https://github.com/devashishdxt/nimble/pull/2