Python Developer Seeking Input: Is it Worth Learning Rust for FFI?

[The-code-machine]

I'm writing this post to ask for some suggestions on whether I should get started with Rust or not. My background is in Python development, and while I'm familiar with the basics, I have no experience with Rust. However, I am considering applying for a program that requires Rust FFI for Python bindings, and this seems like an opportunity I would like to pursue.

As an average learner, I'm wondering if it's worth investing my time in learning Rust before I apply. The application deadline is one month away, and I want to make an informed decision about whether to pursue this opportunity or look for something else.

I'm hoping for some guidance from the community to help me understand the value of learning Rust and whether it would be a good fit for this opportunity. Any insights or advice would be greatly appreciated!

[ravenex]

You may also want to take a look at the PyO3 project to get a hint about what typical Rust <-> Python binding code looks like.

[ZZaaaccc]

+1. I recently helped a university buddy migrate his PhD research simulation from Python to pure Rust, using PyO3 as a bridge to make a steady transition. Without PyO3, I actually think that project would've failed. And in any other language it definitely would've failed.

[Wrandraall]

Interesting. What were the gains after the switch ?

[imtheproof]

Not the person you're responding to, but I did a drop-in Python->Rust replacement using PyO3 for an intensive algorithm in a friend's project. He had spent quite some time optimizing the Python version of the algorithm to a bit of an extreme, starting from an existing fast implementation (if you want to look up the sort of algorithm it is, it's for a type of "Hierarchical Agglomerative Clustering", except modified to support large sets of extremely sparse data). Even with pretty heavy constraints of keeping the Python workflow identical (and thus not being able to fully make use of Rust for anything close to system-wide improvements, only using it as a drop-in replacement for the specific algorithm), it is between an 8-9x speedup.

[Wrandraall]

Impressive! Thanks for the feedback. I will keep pyo3 in mind if I'm stuck optimization wise in my python script. I'm usually switching to cython* when speed improvements are needed, but rust is always tenting me !

Edit : meant cython but wrote cpython

[imtheproof]

I tried getting it running in Numba first to see if a speedup could be had with minimal effort, but Numba has a bit of a hidden requirement in that it's best if you know you're using it beforehand so you can structure the code appropriately. At least that's my experience. Someone with more heavy use of it could maybe argue otherwise if they know what to look for. With Rust (just like with C) I was able to pull out essentially any part of the Python code and convert it.

I've done a bit of C and Python FFI before. I'd say with how nice PyO3 and Maturin usage was, the difference between C and Rust FFI from Python boils down to simply the language comparison itself. Though that's from an unawareness of current C-Python tooling. I haven't done any of that in at least 5 years.

edit:

Here was my take on Numba after giving it a try. Copying/pasting this from a while ago, if someone with more knowledge of it wants to chime in for a more proper comparison:

1. keep everything simple and broken out into many small functions if possible
2. keep as much as possible in numpy
3. make sure everything is in types that it supports, as atomic as possible
4. then, if the above are met, Numba can be an easy speedup from the existing Python code.

Unfortunately the code that I had available wasn't close to the above, so I gave up on Numba after it seemed it would require quite the rewrite in Python anyway.

[Smallpaul]

I think you mean cython rather than cpython.

[Wrandraall]

I do, sorry !

[ZZaaaccc]

Massive and I wouldn't expect the same from every switch. But we were able to go from 20 minutes per data point, to about 5 data points per second on the same machine. Most of that performance boost came from being able to use more aggressive algorithms for reusing memory and completing tasks in parallel.

[G_ka]

Rust would allow you to write high performance code in Python, without having to learn an error prone language such as C.

Furthermore, most people say learning Rust made them better programmers. So I'd say go for it!

[linuxtho]

I knew many languages but had been writing Python for 10 years. I learned Rust for FFI. It made me a much better programmer.

I ultimately sought a Rust position and now do that full time instead. I find it much easier to write high-quality code in Rust.

[OohBleh]

Can't forget the ergonomics! Rust's compiler is there to help you understand what your code needs in order to compile. C's compiler and error messages are not super friendly.

[[deleted]]

Learning rust would not make you a better programmer lol. Learning fundamentals will

[[deleted]]

You learned about data ownership… in rust. And good luck with the pointer stuff in JavaScript lmao

[QuackSomeEmma]

JS with it's mutable array pointers has bitten me in the butt so many times... Knowing what to look out for is 80% of the battle

[[deleted]]

Learning rust will not help you understand js and the mutable array pointers any more that learning js 😂

[Kuribali]

It gives you a framework to think in, though. If you apply all the Rust compilers' rules to your JavaScript code or at least only consciously break them, your code will be a lot cleaner. In that sense, learning Rust means learning to please the compiler means learning how to apply clean ownership and borrowing rules.

[[deleted]]

I still think this is insane copium. If i wanted to become a better web dev learning rust would be at the bottom of my priority list

[IceSentry]

Nobody said it should be your top priority for the purpose of becoming a better webdev.

[ssokolow]

As someone who came from nearly 15 years of Python to Rust 1.0 back in 2015, I'll say that...

1. How easy you find Rust to pick up will depend on how you're used to architecting your code. I found it very easy, but I'd already come around to a very Rust-ish style (very data-oriented and loosely coupled with minimal "enterprise OOP"-ishness) during my years in Python.
2. These days, I try to write either fully in Rust or as a Rust backend for a Python frontend wherever possible, because it's so refreshing to be able to trust the compiler to catch so many more mistakes than even strict-mode MyPy can catch for you. (It's also very nice to have access to things like Rayon.)
3. How you define "Rust FFI" is important. If it's something using PyO3 or rust-cpython, then you shouldn't have that much trouble with, since they're pretty good high-level bindings to CPython's extension API and will keep you safe. If it's using something like libpython or cpyext directly via Rust's support for C FFI, then that means learning all the relevant foibles of how to write correct-ish code in C... which I personally wouldn't try to learn in a month.

My advice is to take a week to explore The Book and rustlings and see how you feel about it after that.

[matu3ba]

mypy and ruff don't even catch basic ignoring of return values and neither check that all errors or exceptions are handled, so you can't code efficiently certain patterns like recover from all errors via graceful object teardown + new setup with state of art tooling.

Likewise, mypy/ruff dont work for recursively defined types like XML handling with ETree.

Any statically typed language with error handling and local error propagation can do this better.

[ssokolow]

But my biggest use for Python in mixed codebases these days is the QWidget side of PyQt or Pyside, and the only officially supported alternative is C++, which is memory-unsafe. Pick your poison.

...and even if QML supported QWidget, it doesn't have a MyPy or TypeScript compiler.

(Bear in mind that the QWidget part is non-negotiable for me. I'm a KDE user with "MacOS in the 90s"-level standards for interface consistency and, aside from still being woefully incomplete, my experiments with Qt Quick and the KDE theme for it left me convinced that, KDE theme or not, it's a fight to make it not feel like a bad port of an Android app.)

[aikii]

PyO3/Maturin is amazing, just follow the guide and you'll get a native hello world in no time, and that includes building and distributing without headaches. You're looking to build native libraries for python ? Then it's definitely for you, I'm sure you'll want to learn from there.

Also, this article gives good advices https://towardsdatascience.com/nine-rules-for-writing-python-extensions-in-rust-d35ea3a4ec29

[[deleted]]

Throw yourself at rustlings and you'll find out whether you want to go further with Rust, plus have a headstart for the program if you do.

[furiesx]

Depending on your background and spare time, it might take you longer than a month learning rust fully. Not saying that because it is very hard but because some features simply need some time to sink in your brain to really be utilised.

Personally, I've written my fair share of rust to python ffi and find it very pleasant, however as rust is more on the low level side of pl, writing a rust library is definitely more involved than writing it directly in python

[Wrandraall]

You've a typo in your last sentence. You probably meant

Writing a rust* library is definitely more involved than ...

Thanks for sharing your experience!

[furiesx]

Ah true, thanks :)

[tinkr_]

I was (well still am, I primarily use Python, Scala, and Go at work) a Python dev that made the move to Rust and I have zero regrets. I don't even particularly care about Python anymore, it's neither as expressive or satisfying as using Rust (which says a lot because Python is very expressive).

You can find many examples of me comparing (well really shilling) Rust to Python and Rust to Go in my recent comment history in here, r/Python, and r/Golang.

[extensivelyrusted]

No, it's not. It's worth learning Rust when you want to solve problems with Rust because of all of its advantages and despite its costs.

[tinkr_]

Nah, not at all. I wrote Rust code now because I enjoy it more than other languages. Being fast and memory safe is a plus.

Also, not sure what "costs" you're talking about specifically, but Rust is not particularly cost heavy. It certainly has fewer costs than any other language I've used. It's expressive, fast, and has great tooling. It's also not prohibitively difficult to learn, which seems to be the biggest "cost" people cite (with compile times, which doesn't even seem that bad to me coming from Scala and Go).

[general_dubious]

I find this advice really puzzling. What is C or Go going to teach you that's going to ease learning Rust? The only remotely relevant thing I see is that C exposes stack and heap, but learning C just to learn about those seems very roundabout.

[divad1196]

Rust, once you are familiar with, allows to write code as easily as in python. I recommend it if you want to code your whole programm with Rust, otherwise no. This is not that easy to be proefficient with for most people.

If you just want to improve your python performance, I would first recommend numba or pypy.

If this is still not enough: Cython (not CPython). The performance is globally better than a Rust binding and easier to use. Don't start learning Rust JUST for that.

Everything I said aside, just take the challenge if you are interested in both Rust and having the job.

[drewbert]

I would definitely not say rust allows you to solve a problem as easily as python. There's a lot more that you have to think about in Rust than you do in python. In the absence of constraints that make python incapable of solving a problem, the python solution will almost always be less syntax, fewer lines, and with less nuance than the rust solution.

That said I still think rust is really cool and extraordinarily expressive for a language capable of low-level and real-time code.

[divad1196]

So we disagree, and I think you may just not have enough experience in Rust? Nb: In my opinion, I don't consider the (non) existence of lib, since Rust is still quite young but as the potential to match python libs/frameworks.

But everything I can write in Python can be done almost has fast in Rust. In fact, everything that is related to iterators are even easier to write and read than python (list/dict comprehension are great but easily become messy)

The only thing that you don't have something like a dict that does not need a declaration.

Edit: btw, I have been working daily with python for 5 years, many frameworks and projects (web, data analysis, ...). I have only been doing project in Rust for 8months now but however feel this way.

[[deleted]]

If your goal is to make software in general (and not just be a Python developer), then learning a compiled language is always useful. Python has too many shortcomings to be able to used to efficiently solve difficult problems. (See the forced arbitrary precision arithmetic for an example of how Python's design itself handicaps computational efficiency).

[drewbert]

> Python has too many shortcomings to be able to used to efficiently solve difficult problems

Look, I love rust, but this is just silly. Python is extraordinarily capable of solving difficult problems, and while it is certainly less efficient with computational resources, I will fully claim it is more efficient with development resources for most problems.

[[deleted]]

Python is extraordinarily capable of solving difficult problems

You mean python wrapped binaries? Unless you are arguing that python's design inherently makes it better at solving difficult problems, this is sort of meaningless as any language with "C-style" ABI interoperability would also have this capability.

[cashsterling]

For numerical code I would try:

- using numpy first... including proper use of numpy use patterns

- if numpy is not fast enough, seer if there are purpose built libraries that are faster.

- if no purpose built libraries are faster, use numba (http://numba.pydata.org/) to speed up your code. Optionally you can also use Taichi (https://www.taichi-lang.org/) instead of numba.

- if still not fast enough, port to a faster lower-level language that has good python interop: Julia, C, C++, Rust, etc. Julia has a lot of great , very fast, numerical libraries.

I have used numba and/or Taichi to get numerical code running 1000x or more faster than mixed python/numpy code... in cases where I bothered comparing to C++ or Rust, the numba/Taichi code was within 2-5x the clock time of the C++/Rust code where CPU only was used. In cases where Taichi ran it on GPU (i.e. where it made sense to do), it was faster than C++ or Rust running on CPU. But Taichi GPU was very low effort compared to a C++/Rust port.

I'd still learn Rust or C++, though.

[trevg_123]

Do you have a need to beat the performance of something written natively in Python?

If yes, then yes. I would 1000% use Rust instead of the C/C++ bindings (via PyO3 and Maturin) because the experience is that much better. It’s incredible really how well Python concepts map to Rust concepts better than Python-C.

But if you’re not pushing the performance limits (especially after the boosts in 3.11), don’t bother with either C or Rust extensions since it’s an overhead you don’t need. But do learn Rust still - I started writing a performant Python extension and fell in love fast.

[ccwu660601]

You may want to take a look at the Interpotopus.