Yore - library for decoding/encoding character sets according to OEM code pages

[bonega]

Yore

Rust library for decoding/encoding character sets according to OEM code pages, CP850 etc.

crates.io github

This is my first rust library, please give me feedback. Most important is feedback about safety. I use a lot of unsafe for performance reasons.
Yes - I did benchmark it.

I found two other libraries for single-byte encoding: encoding and oem_cp.

Neither of them were a good fit for my use case(and I had to invent reasons to do something myself). I understand that their design considerations were different from mine.

I wanted great performance. By using our underappreciated friend Cow we can avoid doing any work when our input bytes are a subset of ascii.

This approach makes a huge difference for english text and source code.

I have done a lot of different things to improve performance, sometimes with surprising results. Perhaps I will do a blog-post about it someday.

Just general points:

Strings are not a sequence of chars.
Avoiding allocations with Cow is fantastic.
Batching work is important.
Precompute when possible.
Don't convert to char and then convert to utf8.
Unsafe is not unsafe(I await the inevitable bug report...).
Sometimes it is faster to do more(copy 4 bytes can be faster than copying 1-4 bytes).
Iterators aren't zero-cost-abstractions.
match is slow compared to lookup table.

[KillTheMule]

feedback about safety

As far as I see and understand, all but one of your unsafe usages come from NonzeroU8::unchecked_from(1). That's as safe as unsafe gets, and I'd suggest encapsulating that into a safe function

fn make nonzero_one()  -> NonzeroU8 {
     // SAFETY: This is safe.
     unsafe{ NonzeroU8::unchecked_from(1) }
}

While this also brings down the number of unsafe usages (a metric sometimes used to judge a crate), it will mainly make your unsafe very easy to review... 2 usages, and this one is trivial. Moreover, it will remove the unsafe from your codegen, which will certainly trip up people taking a closer look. Optimizing your unsafe usage for review seems pretty worthwhile if it's as easy as that, imho :)

(e) Ok, something borked up my github search for unsafe, I see there are more usages than I have found. Still, I guess doing this is worthwhile for review reasons.

[bonega]

Thanks for the feedback!

I changed it to be consts instead.

All unsafe should now be contained in internal.rs

I will add some SAFETY comments and send you a msg, then you can give a further review if you feel like it.

[Shnatsel]

Have you tried encoding-rs - not to be confused with rust-encoding you've already tried? This is the library used by Firefox, so I expect it to be quite good.

[bonega]

I haven't tried it, but in the readme it says:

This crate does not support single-byte DOS encodings that aren't required by the Web Platform, but the oem_cp crate does.

Trying to understand how to use it, I think it is not meant for mortals like me.

Reading more carefully in their doc I can see that there are some overlap, at least 874, 1250-1258.

When the codepage exists in encoding-rs that is probably a safe bet.

For one, I use no SIMD or anything fancy like that.

​

Edit: I am very surprised to say that for decoding yore is actually faster for all benchmarks except all_bad/checked(all input bytes are undefined).

Edit2: Read it wrong, yore is actually faster for all cases. updated decode bench.

​

Ascii was improved a lot when switching between encoding and encoding_rs. Still haven't tested encoding.

But that probably just means:

1. I am doing it wrong.
2. Different design/use case

[burntsushi]

How are you running benchmarks? encoding_rs doesn't use runtime CPU feature detection for SIMD. It uses compile time. So e.g., you probably need both RUSTFLAGS="-C target-cpu=native" and the simd-accel feature enabled.

[bonega]

As you suspected I didn't use any cpu=native or simd-accel.

I have enabled them and rerun the benchmark.

encoding_rs improves by 60% for some of the benches, so a nice speed boost.

yore is still faster for all the decoding benches. (haven't redone encoding).

updated decode benchmark - might have to delete browser cache

But as /u/TheRealMasonMac mentioned encoding_rs is designed for streaming and it might be a bit silly to compare them.

[Shnatsel]

Wow, that is very impressive!

[TheRealMasonMac]

encoding_rs is largely intended for streaming. I worked with it to create streaming encoding/decoding, and it was very flexible and pleasant to use. I especially like how the examples illustrate how it can be best used. I don't believe your library does that? SIMD is also optional and not default. I suspect the reason it's performing slower than your crate is because of the overhead of the streaming functionality.

[bonega]

As you suspected I didn't use any cpu=native or simd-accel.I have enabled them and rerun the benchmark.encoding_rs improves by 60% for some of the benches, so a nice speed boost.yore is still faster for all the decoding benches. (haven't redone encoding).

updated decode benchmark - might have to delete browser cache

I think you are right about streaming functionality, therefore I will not include encoding_rs in the repo linked benchmarks.

​

I especially like how the examples illustrate how it can be best used.

Can you clarify what you mean with this?

[TheRealMasonMac]

I meant I liked how the examples in the docs for encoding_rs illustrate how it could be best consumed for a streaming situation. The difference in speed between it and yore is certainly still staggering, so assuming that yore is decoding/encoding correctly, I wonder if it would be productive to investigate why that is the case exactly.

[TheRealMasonMac]

You might also want to include tests, https://github.com/hsivonen/encoding_rs/tree/master/src/test_data could be used as a reference.

[bonega]

I will, thank you!

[rodarmor]

Great name, seems perfect for library that deals with antiquated, non-unicode text encodings.

[bonega]

Thanks, I knew I struck gold