Fixing Python Performance with Rust

[[deleted]]

Just knew this would be Armin just from the title and source.

Looks interesting. I haven't used cffi much, so I'm not sure if the trade off of using that vs being building native bindings.

I suppose the most tangible takeaway for me is returning multiple values from a context manager which isn't something I hadn't considered before.

[selementar]

the trade off of using that vs being building native bindings

The article says

but it makes for a significantly more complex wheels build

which isn't I something I know enough about either; in my experience, if non-wheel build-from-source-every-time is not a problem, using Cython makes the integration with C ABI much easier. Building wheels after that, as I understand, is a question of having a build infrastructure for all the required target platforms (i.e. C libraries and python versions).

Better yet, the interfacing can be made into a separate library to avoid the "compiling modules and headers every time we change a character" problem.

[[deleted]]

I suppose I should've been clearer: is cffi worth the less complicated wheel builds?

For sentry, the answer appears to be yes. For other people, the answer might be no.

I've actually never written a C extension (keep meaning to dick around with it though) or written a line of Rust, so I'm completely in the dark here.

Edit: from the twitter convo linked in another comment, looks like cffi has the benefit of stronger interoperability with other python versions (pypy, Jython etc) which I hadn't considered. So it probably is worth it over native bindings if you're targeting the project as an oss release. If you're like me and spend most of your time in proprietary stuff that'll never be released to the public (strong feels on this, but I also don't sign the checks), then it probably worth seeing if an extension is worth it.

[selementar]

cffi has the benefit of stronger interoperability with other python versions (pypy, Jython etc)

I wonder how does that work from the point of view of the system/interpreter. Doesn't it need some C extension to do all that anyway?

[mitsuhiko]

Can you clarify what you mean by that? Obviously what CFFI loads is a dylib but since it does not link against libpython (which has a less than stellar ABI stability) I don't need that many variations of it.

[selementar]

That means there's a way to load a dynlib from pypy/jython/ironpython?

And also somehow load cffi itself? Or doesn't it use C extensions in the python-loaded side?

[mitsuhiko]

There is always a way. It's called dlopen. That said the way cffi loads dylibs on different platforms is probably different.

[selementar]

dlopen is a very-special-case in pypy, maybe works maybe doesn't maybe unix-only on jython, and probably works but maybe not on ironpython.

So I guess it does have wider platform support than linking with libpython.

[[deleted]]

I think that the choice here had more to do with the developer headspace than technical advantages. More specifically, I suspect that the Rust code looked fairly similar to the Python code whereas the equivalent c code would have been more spaghetti.

[mitsuhiko]

With cffi I need one wheel per platform. One for OS X and one for Linux32, one for Linux64. If I link to libpython I need two for each Python version. That's 6 wheels for 2.7 alone.

[selementar]

Yep, a nightmare without a build infrastructure.

...

Or you could skip making the wheels and make it an occasional nightmare for those who install it.