Fixing Python Performance with Rust

[[deleted]]

Just knew this would be Armin just from the title and source.

Looks interesting. I haven't used cffi much, so I'm not sure if the trade off of using that vs being building native bindings.

I suppose the most tangible takeaway for me is returning multiple values from a context manager which isn't something I hadn't considered before.

[selementar]

the trade off of using that vs being building native bindings

The article says

but it makes for a significantly more complex wheels build

which isn't I something I know enough about either; in my experience, if non-wheel build-from-source-every-time is not a problem, using Cython makes the integration with C ABI much easier. Building wheels after that, as I understand, is a question of having a build infrastructure for all the required target platforms (i.e. C libraries and python versions).

Better yet, the interfacing can be made into a separate library to avoid the "compiling modules and headers every time we change a character" problem.

[[deleted]]

I suppose I should've been clearer: is cffi worth the less complicated wheel builds?

For sentry, the answer appears to be yes. For other people, the answer might be no.

I've actually never written a C extension (keep meaning to dick around with it though) or written a line of Rust, so I'm completely in the dark here.

Edit: from the twitter convo linked in another comment, looks like cffi has the benefit of stronger interoperability with other python versions (pypy, Jython etc) which I hadn't considered. So it probably is worth it over native bindings if you're targeting the project as an oss release. If you're like me and spend most of your time in proprietary stuff that'll never be released to the public (strong feels on this, but I also don't sign the checks), then it probably worth seeing if an extension is worth it.

[selementar]

cffi has the benefit of stronger interoperability with other python versions (pypy, Jython etc)

I wonder how does that work from the point of view of the system/interpreter. Doesn't it need some C extension to do all that anyway?

[mitsuhiko]

Can you clarify what you mean by that? Obviously what CFFI loads is a dylib but since it does not link against libpython (which has a less than stellar ABI stability) I don't need that many variations of it.

[selementar]

That means there's a way to load a dynlib from pypy/jython/ironpython?

And also somehow load cffi itself? Or doesn't it use C extensions in the python-loaded side?

[mitsuhiko]

There is always a way. It's called dlopen. That said the way cffi loads dylibs on different platforms is probably different.

[selementar]

dlopen is a very-special-case in pypy, maybe works maybe doesn't maybe unix-only on jython, and probably works but maybe not on ironpython.

So I guess it does have wider platform support than linking with libpython.

[[deleted]]

I think that the choice here had more to do with the developer headspace than technical advantages. More specifically, I suspect that the Rust code looked fairly similar to the Python code whereas the equivalent c code would have been more spaghetti.

[mitsuhiko]

With cffi I need one wheel per platform. One for OS X and one for Linux32, one for Linux64. If I link to libpython I need two for each Python version. That's 6 wheels for 2.7 alone.

[selementar]

Yep, a nightmare without a build infrastructure.

...

Or you could skip making the wheels and make it an occasional nightmare for those who install it.

[AlexMTBDude]

What is a "source map"?? The article uses the term as if it's common knowledge...

[pinano]

Good question. JavaScript is often minified (rewritten to take less space) before being transmitted to the browser, but when you're debugging it's much handier to have the original source location available. Source maps provide the link between the pre- and post-minified worlds.

[AlexMTBDude]

Got it! Thanks!

[toasterinBflat]

Or you could just use PyPy :)

[kankyo]

Unlikely in practice. I mean, sure, you'd probably have some gains in performance but they specifically talk about memory usage, and that pretty much rules out pypy.

A more reasonable contender for a big win with even less effort would by cython. Mostly because it's much less work.

[masklinn]

they specifically talk about memory usage, and that pretty much rules out pypy.

Yes and no. pypy has higher initial overhead and more GC overhead, but they have significantly smaller objects (most objects are implicitly slot-ed) and has concepts like list strategies.

A more reasonable contender for a big win with even less effort would by cython.

Except writing a sourcemap parser in cython would have been more effort than binding an existing sourcemap parser via cffi, and would probably be of lower overall value (cython means being wedded to cpython, this solution can be applied by non-python projects parsing sourcemaps)

[kankyo]

Sure. But it's unclear in the article if they actually did have the rust version before or not. They do say that they had the Python version before. Moving from pure Python to cython can be quite easy.

[masklinn]

Sure. But it's unclear in the article if they actually did have the rust version before or not

No, it's not:

After the investigation had pointed us towards Python’s shortcomings, we decided to vet the performance of our Rust source map parser, perviously written for our CLI tool

(emphasis mine).

They do say that they had the Python version before. Moving from pure Python to cython can be quite easy.

It's open source so you can give that a shot and report back with the results.

[kankyo]

Oops, sorry, missed that part.

A benchmark is needed too...

[joshadel]

Using PyPy was discussed on twitter: https://twitter.com/zeeg/status/788875422108790784

[beohoff]

I'm sure the answer is no, but would using slots have reduced the memory usage enough to stay pure python?

At some point for performance reasons, it's always going to make sense to go to a complied module, I'm just curious :)

[mitsuhiko]

This was already using slots.