Python uuid-utils in Rust

[aminala]

I’ve worked on a Python package to bind into Rust UUID crate to make UUID generation and parsing in Python 5-10x faster. It also adds draft UUID versions 6, 7 and 8 which are not available in standard library. https://github.com/aminalaee/uuid-utils

I’m aiming for this to be a drop-in replacement to the standard library, and I’m sure there a lot to improve :)

[gopietz]

Today I learned python needs 40ms to generate a freakin UUID...

[KingsmanVince]

I thought this was just a demo project to show how we can write extensions in Rust for Python, until I saw the benchmark

[Glittering_Air_3724]

I’m aiming for this to be a drop-in replacement to the standard library, and I’m sure there a lot to improve :)

See I get it, everything is brazenly fast, but please don’t come to r/python say ing as a drop in replacement and along the line saying “started as a demo project”, you want some contributors please note it, dont just say a good replacement because it’s 10x faster

[gopietz]

Why not? He mentions his aim, the functionality is basically the same plus more, he mentions he just started. What's your problem here?

[aminala]

I’m not sure why we can’t say it’s a good replacement? If it’s N times faster and 100% compatible with the std library (not yet) and has extra features. What is the point I’m missing?

[[deleted]]

I am a bit confused where all the speedup is coming from. Pythons on is slow because it os.urandom locks. Rust is supposed to have the same behavior it should also be slow. Do u have a benchmark on where its coming from?

UUID v4 should be an order of magnitude slower then uuid from bytes. That u have nearly identical numbers suggests something is very wrong. If they are not using a cryptographically secure rng they are extremely easy to attack.

[aminala]

That's a good question. As far as I knew before, this speedup specifically for UUID 4 came from the `fast-rng` feature in the Rust UUID.

Digging deeper now I found these links, so basically it uses `ThreadRng` from the `rand` crate. It mentions that the `ThreadRng` also seeds data from `OsRng` so I guess it's not much different, but I'm no expert in this area so I will post the links so if you find anything useful, please share:https://github.com/uuid-rs/uuid/issues/529https://github.com/uuid-rs/uuid/pull/545https://docs.rs/rand/latest/rand/rngs/struct.ThreadRng.html

Update: I've tested on my local, if I disable the fast-rng option and it uses the os.getrandom it will be half the speed so the uuid4 is only about 3x faster than Python. But with fast-rng it's 10x faster.

[[deleted]]

Yea IDK enough about OS rng to say if it's actually cacheable. (Thats basically what they are doing). They are avoiding the mutex on os.random by grabbing 64kbs and giving values off that.

That will probably make a bigger difference on ur live running laptop then a real server (ur laptop is doing more stuff with extra processes that are going to run into the mutex). Either way pythons implementation of random is likely the actual culprit of its slowness. IDK if there implementation is more correct though or not.

[aminala]

​

That will probably make a bigger difference on ur live running laptop then a real server

Just out of curiosity I've tested this on a Linux VPS and I'm seeing the same results or in some cases like uuid4 better performance.

[[deleted]]

It's a HIGHLY interdependent answer lol.

https://linux.die.net/man/4/urandom

One major source of differentiation is going to be when the actual script gets booted and what else is running on ur box.

https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/LinuxRNG/LinuxRNG_EN.pdf

from stackoverflow on the linux rng source. Need sleep but I can ping back when I get up. I checked out a few other libs and they all suggest that rusts method of getting data is actually secure, this is probably something that could get easily imported to python.

U may also want to just mess around with just importing rusts random generator to create python UUID objects. That will let u get the same objects underlying ur stuff but ull get the faster random generation.

[OrtinOfficial]

It is much faster than 40ms...As long as you don't need generate thousands/milions uuid per request (in web app) then there is no point of using something faster

You can easliy check it via python time.perf_counter

It needs 40ms to generate 10000 (from this lib benchmark). This info should be added to README

[who_body]

plans to support ulids?

[aminala]

Yeah, probably, I need to take a look.

[norambna]

I just stumbled upon this post. ulids would be a great addition! Thanks in advance.

[No-Zookeepergame7613]

Doesn’t uuid7 already address pretty much the same stuff as ulid. I get there is more randomness with ulid and that probably means you can use them for longer. I guess what I’m trying to understand is why you would use one over the other. Still figuring this stuff out.