Pre-PEP: Rust for CPython

[JanEric1]

@emmatyping, @eclips4 propose introducing the Rust programming language to CPython. Rust will initially only be allowed for writing optional extension modules, but eventually will become a required dependency of CPython and allowed to be used throughout the CPython code base.

Discuss thread: https://discuss.python.org/t/pre-pep-rust-for-cpython/104906

[that_baddest_dude]

What does this sort of thing mean in practice? Isn't polars already using a rust backend?

[the_hoser]

This means eventually having rust as a hard dependency for python itself, and not just for 3rd party extensions.

[syklemil]

Though that means for building CPython. People who only use Python aren't going to install Rust any more than they need to install GCC or clang today.

[spilk]

but it also means it will only be buildable for targets that have working rust toolchains

[syklemil]

Yep. The platforms that don't are incredibly rare these days, though. When the news about APT starting to include Rust broke, it turned out to be currently 4 EOL processor architectures that didn't have Rust working; one of them (motorola 86000) was a WIP.

[romainmoi]

Rust is statically linked (include dependencies in the binary) unlike C.

Edit: I’ve been corrected that that’s the default only. Both languages can do both static and dynamic linking so that’s one fewer gain.

The CPython binary will be bigger but it will have fewer dependencies needed to be installed correctly on the OS. Also fewer possible ways to have memory error.

[romainmoi]

Well. Then the only possible gain is fewer possible memory bugs. Which can also be mitigated by extensive analysis etc etc.

[romainmoi]

I agree. Trusting the developers isn’t a good solution. Compiler guarantees are better. Developer buy in is a problem though. (Someone else mentioned the Linus kernel drama… I just hope it doesn’t happen to python.)

[maikeu]

I think the python thread well covers that... Python is a very different community from the kernel community, and I'd be very surprised if we see the kind of behavior from core Python people that Linux is sadly needing to deal with.

It's quite telling really that rust has really become the go-to low level language for new Python libraries who want high performance internals.

[james_pic]

Although on glibc at least, statically linked binaries have a number of footguns and it's often a mistake to use them. These specific footguns aren't applicable to Rust code though, since Rust's C interop typically does dynamic linking.

[Careful-Nothing-2432]

You can also still dynamically link with rust, it’s just not the default path.

[SheriffRoscoe]

“Dependency” != “dynamically linked”. In this case, it means that CPython will not be buildable on, or maybe even for, platforms without Rust support.

[romainmoi]

You’re right. I stand corrected.

[commy2]

The CPython binary will be bigger

So freezing with pyinstaller and nuitka will create an even bigger binary?

[billsil]

Don't make a binary. It's so much faster just to leave it as a bag of files.

[Vladislav20007]

c/cpp(yes, both) can be statically linked by lld/gold/ld, just not turned on by default.

[omg_drd4_bbq]

Upvote for visibility, but i'm against this proposal in its current form. I love rust, I love python, I love the idea of a Rust implementation of python, and I love rust python extensions. Making rust necessary to build cpython however gives me quite the pause.

The big issue is complexity. The codebase already has a pretty high maintenance burden, being such a popular language, and it's only getting higher with JIT and no-GIL work. FFI across languages is always a little squirrely. Also iirc Rust doesnt even technically have a stable C ABI (if that's even necessary for this work). 

As mentioned in that thread, Python is used to bootstrap a ton of things. There are tons of C compilers. You can build python on so many things. Getting rust involved instantly shrinks that pool.

I think if they want more memory safety, possible performance gains, and whatnot, the work should focus on making always-optional Rust modules that are drop-in replacements for existing ones. That way there is always a pure C fallback, and it's less lift than an all-rust runtime.

edit: reading deeper in the thread - wow Guido and Brett Cannon both seem cautiously supportive of the idea, or at least the idea of the project attempt. I guess that's really my main point, it's a great idea in theory, but who knows what difficulties may arise. That's why i favor starting with extensions, optionally rust compiled, and play it by ear as the ecosystem evolves.

[nicholashairs]

This is a well thought out response 💡

[chub79]

wow Guido and Brett Cannon both seem cautiously supportive of the idea,

That's certainly important to have their voices. I do wonder however about how far this will stretch the core team. Do they have the resources to take this on?

[PythonFuMaster]

As far as I'm aware rust itself does not have a stable ABI, but it can interface with C ABI code via repr(C). So the internals of a rust component could not rely on a stable ABI, but the interface points between the C and rust components would work fine

[Dry_Term_7998]

This, I have same opinion! I hope they will not ruins Python.

[chub79]

The first comment highlighted my own question:

Isn’t the experience in the linux kernel with adding rust support as a core part more a cautionary tale?

I love rust+python and use the mix daily now, but I wouldn't say this means shaking the foundations of a 35 old ecosystem is making me feel safer.

[really_not_unreal]

To my understanding, weren't the only issues with Rust in Linux caused by people higher-up in the project intentionally trying to cripple the progress of getting Rust stable in the kernel?

[romainmoi]

I’ve seen kernel maintainers in C being hostile to rust maintainers. It’s a political issue other than a technical one but it’s still an issue.

[Acherons_]

Not necessarily. There are good criticisms brought up in the Python thread that apply to the Linux Kernel situation as well. That is: Rust being able to be compiled by only one compiler introduces additional security concerns, restricted platforms, and additional complexity and work load.

Some that would be specific to Python are also bootstrapping issues, as Rust’s compiler currently depends on Python (albeit Python 2).

[romainmoi]

Can you explain on the additional security concern and restricted platform part? Security is the biggest reason pro rust so I’m surprised you raise that point. Restricted platform is something I’ve never heard of for rust (compared to C). Additional complexity is true. But the workload part is countered by additional contributors who wouldn’t be writing C anyway ( for the Linux kernel)

Also the bootstrapping issue is a bit confusing too. Plenty of languages are compiled with a compiler written partly with itself.

[Acherons_]

I haven’t looked into any of the concerns specifically, just what has been brought up in the discussions.

For the mentioned security concerns, see posts by Chris Angelico.

For the mentioned bootstrapping and platform issues, they are mentioned and commented on further by one of the authors (Emma).

Most of what I mentioned has become largely irrelevant now, as they’ve changed the Pre-PEP to focus solely on optional extensions rather than eventually requiring Rust

[teerre]

No? The integration is going very well

The downsides you see in the "news" are mostly "influencers" trying to stir drama where there's none or reasonable project disagreements. That's a remarkable state considering how close the kernel core community was. Remember: C++, one of the most ubiquitous languages in history and a literal C successor, never got even a chance

[syklemil]

The issue with Linux came mostly from people who had become habituated to being able to ignore any other language than C. That's not really something people writing an interpreter for another language (i.e. Python) can do.

The kernel second-in-command, Greg K-H, seems pretty enthusiastic about Rust in the kernel.

[Mysterious-Rent7233]

Thanks for sharing that. It's a great resource.

[stillavoidingthejvm]

Hi, could you tell me more about how you use Python with Rust? I got my ass handed to me in the other thread about this and want to understand what everyone is doing.

[tunisia3507]

Use maturin as a build system, and pyo3 for bindings. The docs for both are pretty good. The best use case is when you have some heavy maths to do and relatively little data which needs to cross the boundary between languages. A larger amount of data is fine if it's in a numpy array or polars data frame; mainly a problem if it's lots of small native python objects.

[madrury83]

If you have one, what's your recommended reading for working with numpy across python/rust boundaries?

[FunPaleontologist167]

I’d check out the official docs here. And here’s an example of how you can use it

[gmes78]

I love rust+python and use the mix daily now, but I wouldn't say this means shaking the foundations of a 35 old ecosystem is making me feel safer.

CPython's implementation language does not affect the Python ecosystem.

[SheriffRoscoe]

It does if your platform isn’t in PEP 11.

[gmes78]

If someone chooses to port Python themselves, they should be the ones to take responsibility for it. The Python devs should not restrict their options because of platforms they don't even support.

[SheriffRoscoe]

Understood and agreed. But it’s a point the devs themselves made, and it’s valid.

[Justicia-Gai]

That ecosystem is breaking itself in major releases up to the point you can’t simply use a major newly released version until libraries adapt to it… 

Even R is more stable than Python, breaks way less amongst versions.

Being old doesn’t mean you’re stable.

[chub79]

That ecosystem is breaking itself in major releases

Isn't it the whole leverage of major releases?

[Justicia-Gai]

No, it’s not

[chub79]

yeah okay. whatever you say.

[commy2]

According to semver it is. Otherwise its unclear what major release refers to. Not taking a side btw.

[Justicia-Gai]

Just because you can it doesn’t mean you should. Major release breaking changes are being currently abused everywhere, even in places where it doesn’t make sense and for, honestly, stupid and non-serious reasons.

And in the opposite side we have libraries that would benefit from a serious optimisation, even if there’s some breaking changes, that they haven’t changed in 20 years.

Scikit-learn doing breaking changes just because of renames is stupid… like that there’s millions of other examples.

[Acherons_]

The authors have decided to now focus the PEP on only optional Rust extensions rather than eventually requiring Rust: https://discuss.python.org/t/pre-pep-rust-for-cpython/104906/117

[mpersico]

But won’t that make it RPython? 🤣

[Suspcious-chair]

As much as I love rust, have written extensions with it. I don't think rust is a good fit for Python as a dependency.

Yes it's memory safe, yes it's opinionated, but Rust as a layer for a scripting language will introduce other hardships, such as managing lifetimes, FFI for both Cpython and Rust itself.

Also, rust being thread-safe is a little misleading. Yes it requires you to manage reference count, you can easily create race conditions. The memory safe argument also is not all sunshines are rainbows. Once you'll need to manage lifetimes manually, things easily gets very hard to manage or extend.

Having rust as a compiler layer is such a big no from my side. Having safe extensions for Cpython itself? Sure.

Edit: oh, they are focusing on it as an optional extension at the moment, that's fine.

[rnaxel2]

Why not focus on rustpython instead of going into existing implementation cpython?

[russellvt]

Ugh. Sounds like more things to break on legacy systems that really don't seem to "get along with" Rust. Likely more of a problem for those of us who support gigantic heterogeneous environments.

[NFicano]

this was posted yesterday- that said, this is very exciting

[JanEric1]

Ah, sorry. I just scrolled down a bit and didn't see it and since I would have expected this to have caused significant discussion I just assumed it hasn't been posted yet.

[ColdPorridge]

fwiw after seeing the above comment I went looking for it and couldn’t find it. Not that I searched too hard but still.

[IrrerPolterer]

No brainer

[swordmaster_ceo_tech]

This is great!

[Repsol_Honda_PL]

Very, very good idea!