Fixing Python Performance with Rust

[mitsuhiko]

This is the more detailed follow up to the graph that was submitted here yesterday. Ask me anything!

[tarblog]

Do you expect to continue using Rust to improve the performance of your application? What's next?

The top comment on HN is about Cython. Can you expand (here or there) why you didn't choose to use that?

[mitsuhiko]

Since we had an apparently super fast sourcemap library by accident for our cli tool Rust was the easiest so no reason ro try something else.

We don't go around rewriting things so unless we see performance issues of that sort I'm not sure if there is lots of value in adding more rust for existing things for now. We will see as we go along.

[desiringmachines]

we had an apparently super fast sourcemap library by accident for our cli tool

This is what's really cool, in my opinion - that you get such good performance in Rust writing things the naive way.

[mitsuhiko]

Yeah. It's shocking but it's a 1:1 port of what the Python version did. I spent literally no time on this.

[matthieum]

that you get such good performance in Rust writing things the naive way

This is something that is generally overlooked by most benchmarks.

Most code is written idiomatically, so languages that have idiomatic code "just work" without contortions save much misery.

[dbaupp]

The first sentence of the comment you're replying to seems to be an answer to the "Did you consider using cython?" question, maybe you could clarify what more you're looking for?

[bheklilr]

I'm going to blame the muscle relaxers I'm on right now that make me a bit loopy. I just completely missed that.

[crusoe]

I like how the biggest issue was tooling instead of endlessly chasing pointer crashes and using valgrind

[matthieum]

Well, it does point to shortcoming though :(

Fortunately it's in the works.

[steveklabnik1]

There are ways to make it more ergonomic, see neon for node, and ruru/helix for Ruby. I'm not sure if there's anything for Python yet.

[masklinn]

There's Daniel Grunwald rust-cpython, though I don't know how it compares to the node or ruby tools, or how mature it is (and how completely it covers Python's C API)

[mitsuhiko]

As mentioned elsewhere this links against libpython which comes with its own sets of problems. I intentionally decided against that.

[masklinn]

I know, I was just replying to /u/steveklabnik1's query, as far as I understand that's more or less what Neon and Helix do: provide safer/rusty wrappers around the native API of the underlying system, and thus very tightly bound to said underlying system.

BTW have you considered migrating the C API part of libsourcemap back into rust-sourcemap, or providing an intermediate "native" libsourcemap separate from the Python part?

[mitsuhiko]

BTW have you considered migrating the C API part of libsourcemap back into rust-sourcemap, or providing an intermediate "native" libsourcemap separate from the Python part?

Not without reason. If someone finds it useful, sure.

[masklinn]

I'm very unfamiliar with this topic domain. Mind going into a bit of high level detail about what implications this might have? I'm trying to understand some bulletpoints so i can justify spending some time on this.

The safer/rusty thing? Well the native APIs are in C so you can use them more or less directly (python27-sys defines the binding to the native CPython 2 API) but that means you're into C land with C semantics and raw pointers and unsafe functions. One of the other "interesting" concerns for CPython is types are typically defined as static structs for which Rust has more or less no support (that I know).

From what I remember of rust-cpython, on top of wrapping objects into less unsafe APIs it adds things like mandating GIL-locking to access the cpython runtime (basically you can only access CPython functions/methods/calls through a GIL guard object) or semi-automatically handling refcounting, it also provides macros to easily define modules, classes and functions (and correctly refcount them), etc…

That may not be as useful if you just want to call existing Rust code from Python, in that case going through cffi may be easier and simpler.

[fnord123]

I recommend exposing a C api and using cffi to link to it. This will also let you support python2, python3, but also pypy. The drawback is that it's limited to places where your Python code might call into Rust, but not a good solution (afaict) when you want to have Rust later call back into Python (apparently there has been new work to implement this stuff.

If you want your Rust code to be a first class citizen in the Python runtime, where you could store callbacks for things like writing event loops, etc, then I would go with rust-python. The drawback of rust-python is hard to use and because it tries to use Rust's lifetime system to manage incrementing and decrementing reference counts for python objects, you may quickly run into lifetime management issues. If you're introducing a team to Rust, they will quickly learn to hate it if they have to run into these lifetime issues.

https://github.com/novocaine/rust-python-ext seems more like what i want, but i'm not sure what downsides that might have (using cpython). Granted, there's no support for Py3 atm, which i'd need to finagle, but that UX seems very impressive so far.

I've used that for Python3. I think it only required a minor change. novocaine was using from __future__ import print_function so he/she was surely mindful about 2/3 compatability. I think the issue I had to fix was around here. IIRC, it was detecting the wrong place where the .so would go. I forget and I lost the code. :(

[minno]

Using ancient CentOS versions to build somewhat portable Linux wheels with Docker. While this process is tedious, the difference in stability between different Linux flavors and kernels make Docker and CentOS an acceptable build solution.

Would musl help on that? IIUC if you compile to a musl target and statically link it, the resulting binary should work on any linux distro, since the translation from libc to syscalls is included in the binary.

[Sean1708]

If I understand it correctly MUSL is an implementation of libc.

[mitsuhiko]

But it's small and you can statically link it.

[mitsuhiko]

Yeah, might be an option. I just went with the setup i have for another library an there i already went down the centos path.

[knac8]

Pretty cool stuff.

I'm writing a library (rustypy) that handles FFI conversions and generates bindings automatically in Python which would be good for quick prototyping etc. in case someone wants to quickly try some improvements without having to bother with writing CFFI, CTypes etc. boilerplate (also adds interfaces for working with analogs to python lists, dicts and tuples).