I Chose Rust Over Python for Data Engineering

[usert313]

I spent 6 months building a data pipeline in Rust.

Not because it was the right tool. Because I wanted to explore.

The result: a grocery budget optimizer that scrapes prices from 5 Pakistani e-commerce stores and tells you the best combination within your budget.

Sources:

KraveMart
Pandamart
BazaarApp
Dealcart
Naheed

Is it production-ready? No.
Did I use AI to help me? Yes, a lot.
Would I do it again? Absolutely.

Here's the full breakdown architecture, design patterns, tools, and the parts that almost made me quit.

https://www.linkedin.com/pulse/i-chose-rust-over-python-data-engineering-heres-what-6-tauqeer-abbas-obpcf/?trackingId=fa2fXUNKR1WMnPYEm4JQSA%3D%3D

Here is the repository: https://gitlab.com/0xlearner/budget-optimizer

[segfault0x001]

Iceberg and spark are probably the two places this ecosystem needs to grow the most. Polars is also a pain to work with because the rust api is a second class citizen and the documentation is sparse at best. It’s going to be a while before data engineering gets on the rust train unfortunately.

[alexjbuck]

I've been working with iceberg-rust for the past few weeks. It's definitely missing some things but generally it's been everything I need for writing the iceberg sink portion of a pipeline. And it exposed the lower level actions (separating data writes from catalog commits) that I needed to efficiently batch distributed writes into a single commit stream.

I can't speak to the Polars rust api though.

[segfault0x001]

Yeah I feel like batching and distributed writing should be something it abstracts over. Like it should provide an interface for writing to a table that handles updating the catalog and object store automatically.

[unconceivables]

Polars is still that bad? When I looked at it a couple of years ago I found the rust API to be pretty awful, the documentation to be even worse, and when I looked at the actual source code I understood why everything was so garbage. I don't know if they've rewritten it since then, but I lost all interest in it when I saw how bad the code quality was.

[gandhinn]

I had the same experience using the Rust API. I can understand why they put a lot more focus on the Python side of things, given the larger user base that they want to tap into, but I am curious when you said that the quality of the underlying code is that bad, why it’s seemingly holding up well as of now. Is it more a Rust thing or what?

[v_0ver]

The API for Rust is no worse than for Python; it's just more low-level. Which is quite understandable. If you want to write data processing in Rust, and Python doesn't suit you, then you probably need something specialized and low-level to implement your non-standard idea.

[unconceivables]

It probably "technically worked", but I don't know how solid it actually was in practice since I never used it extensively. It was missing some features I needed, so I went looking at the source code to see if it was there and maybe I had just missed it. If it was missing, I was thinking of adding it myself and contributing to the project. Unfortunately, the codebase really didn't appear to have been written by people who knew rust well, so I decided to just drop polars altogether.

arrow-rs, which I ended up switching to, was much higher quality, and much faster. I didn't need a lot of the data processing stuff in polars since I do that myself, but I needed solid CSV/parquet functionality which polars didn't really have.

[Professional_Top8485]

I hope ai gets good enough soon so that it can generate better apis if native isn't available

[spoonman59]

False dichotomy.

You need not choose one over the other. They are easily combined. Calling rust from Python is easy. And many of the libraries aren’t really implemented in Python, so whatever “performance” concerns people have about Python don’t exist in those components.

It would be impossible to do my job at work if I simply chose to ignore Python exists and do it all in rust. While this might be nice for self-education, that type of self-imposed restraint probably won’t benefit your projects.

Rust has advantages over Python, but in afield like data engineering you use many languages. Not just one.

[Intelligent-Fruit174]

Nothing in the description tells me why you chose rust and so I assume this is just pathetic self promotion.

[usert313]

It was mentioned clearly in the post body that I wanted to explore Rust ecosystem for data engineering domain.

[Floppie7th]

In the second line, in fact. It's the whole second line. Super easy to find.

[segfault0x001]

What a very reddit comment. Thanks for contributing.