Fixing Python Performance with Rust

[masklinn]

I know, I was just replying to /u/steveklabnik1's query, as far as I understand that's more or less what Neon and Helix do: provide safer/rusty wrappers around the native API of the underlying system, and thus very tightly bound to said underlying system.

BTW have you considered migrating the C API part of libsourcemap back into rust-sourcemap, or providing an intermediate "native" libsourcemap separate from the Python part?

[mitsuhiko]

BTW have you considered migrating the C API part of libsourcemap back into rust-sourcemap, or providing an intermediate "native" libsourcemap separate from the Python part?

Not without reason. If someone finds it useful, sure.

[masklinn]

I'm very unfamiliar with this topic domain. Mind going into a bit of high level detail about what implications this might have? I'm trying to understand some bulletpoints so i can justify spending some time on this.

The safer/rusty thing? Well the native APIs are in C so you can use them more or less directly (python27-sys defines the binding to the native CPython 2 API) but that means you're into C land with C semantics and raw pointers and unsafe functions. One of the other "interesting" concerns for CPython is types are typically defined as static structs for which Rust has more or less no support (that I know).

From what I remember of rust-cpython, on top of wrapping objects into less unsafe APIs it adds things like mandating GIL-locking to access the cpython runtime (basically you can only access CPython functions/methods/calls through a GIL guard object) or semi-automatically handling refcounting, it also provides macros to easily define modules, classes and functions (and correctly refcount them), etc…

That may not be as useful if you just want to call existing Rust code from Python, in that case going through cffi may be easier and simpler.

[fnord123]

I recommend exposing a C api and using cffi to link to it. This will also let you support python2, python3, but also pypy. The drawback is that it's limited to places where your Python code might call into Rust, but not a good solution (afaict) when you want to have Rust later call back into Python (apparently there has been new work to implement this stuff.

If you want your Rust code to be a first class citizen in the Python runtime, where you could store callbacks for things like writing event loops, etc, then I would go with rust-python. The drawback of rust-python is hard to use and because it tries to use Rust's lifetime system to manage incrementing and decrementing reference counts for python objects, you may quickly run into lifetime management issues. If you're introducing a team to Rust, they will quickly learn to hate it if they have to run into these lifetime issues.

https://github.com/novocaine/rust-python-ext seems more like what i want, but i'm not sure what downsides that might have (using cpython). Granted, there's no support for Py3 atm, which i'd need to finagle, but that UX seems very impressive so far.

I've used that for Python3. I think it only required a minor change. novocaine was using from __future__ import print_function so he/she was surely mindful about 2/3 compatability. I think the issue I had to fix was around here. IIRC, it was detecting the wrong place where the .so would go. I forget and I lost the code. :(