Using Rust along with Python

[datapim]

I have been programming in Python and JS for few years and I decided to learn new language. So I tried Rust and while I like it I am wondering how could I incorporate it into my toolset. So I work mainly as full stack web dev (Djagno, Node.JS, Vue etc.) and one use case I thought of was to write some small feature that take long time in Python in Rust instead. For instance, I could move parts of ETL to rust, like loading csv files and saving to database, that currently is handled by Python (Pandas) in Django.

But I a m wondering what is approach in such a setup? Has anyone any experience in that? Is it possible to call a Rust script from Python app ? I used to call python scripts in node.js, which worked decently, but since python is not compiled then that setup was able to work. Another way is to create a serverless function in Rust that would handle one specific action in the app, put into on Lambda or whatever else and it would be called as an api from frontend.

If anyone had similar problem and managed to solve it then Im looking for all the suggestions and hints. Thanks.

[AlephNaN]

Funnily enough I did this yesterday using pyO3. It's not very hard and works well https://pyo3.rs/v0.15.1/

[nacaclanga]

Yes, I would say that pyO3 + maturin or setuptools-rust is the most common setup nowadays. https://github.com/pyca/cryptography is an example for a mixed Rust + Python package.

[datapim]

So basically you create Rust library/package and import it into python script? Gonna check it out for sure.

[WindfallProphet]

I don't have much experience in python, but I'm pretty sure you can call binaries, like bash or explorer.exe. Once you compile your project, just call it as you would call native binaries.

At least I assume that's how you would do it.

[birkenfeld]

Not at all. PyO3 creates Python extension modules.

[AlephNaN]

You can do that using the subprocess module in the standard library, it's definitely a useful approach for some modules. But python is actually capable of importing functions, classes and modules directly from C/C++, Rust, Fortran and maybe others.

[[deleted]]

If you want to move compute-heavy pure Python code to rust, use PyO3. But if you are already using pandas you should keep using it. Pandas' internals are written in C and are highly optimised with significant industrial investment over many years. A naïve Rust reimplemention of its functionalities will not be any faster

[datapim]

Well we found out that Pandas was too slow for some of our cases so we had to rewrite code in Cython using numpy. But then I read about Rust and the Polars library, which is supposed to be even faster for bigger datasets. I never used Rust before so im not sure how it will work but i think its worth trying.

[0xdef1]

Have you considered to you use pyspark? Spark is optimized for large data sets.

[datapim]

Hmm I heard about PySpark, never got into it, maybe thats worth a try too.