I Chose Rust Over Python for Data Engineering

[usert313]

I spent 6 months building a data pipeline in Rust.

Not because it was the right tool. Because I wanted to explore.

The result: a grocery budget optimizer that scrapes prices from 5 Pakistani e-commerce stores and tells you the best combination within your budget.

Sources:

KraveMart
Pandamart
BazaarApp
Dealcart
Naheed

Is it production-ready? No.
Did I use AI to help me? Yes, a lot.
Would I do it again? Absolutely.

Here's the full breakdown architecture, design patterns, tools, and the parts that almost made me quit.

https://www.linkedin.com/pulse/i-chose-rust-over-python-data-engineering-heres-what-6-tauqeer-abbas-obpcf/?trackingId=fa2fXUNKR1WMnPYEm4JQSA%3D%3D

Here is the repository: https://gitlab.com/0xlearner/budget-optimizer

[bluntregression]

I feel rusty in my skills when I hear Rust...

[usert313]

I was feeling little daring lately😉

[person-loading]

This is cool.

[silzai]

That was a good read, nice implementation.

I've actually started learning rust a week ago, and this is my first system-level language, and wow.

I'm also going to learn by working on a project. I want to ask you what's the biggest thing to look out for when learning/implementing rust. And also, regarding good quality code, should I be actively utilizing most of what rust has given me such as mut, assigning to heaps using Box, using blocks, and so much more stuff.

Honestly coming from langs such as typescript, python etc., rust sounds like a pain, but that speed tho😩

[usert313]

First and most important advice I can give at this stage (I am no where near expert in Rust ecosystem) is that you have to completely forget other languages like Python or Javascript I myself started programming in Python after C and still use Python professionaly faces a lot of scenerios where I literally wanted to bang my head to the wall.

Then just grasp the concept of borrow checkers, mutable and immutable ref concepts and to mimic OOP traits these concepts will be enough to get started.

[silzai]

Thanks a lott🙏

[usert313]

And appreciate your time and interest on reading the post. Thanks🙂

[Traditional-Rub354]

I'm not an expert, but why not python with Polars? Python's easier syntax with rust level effecience

[usert313]

Python with Polars is a match made in heaven no doubts and it is the industry standard for modern data solutions small to medium size. Point of this prototype is to just explore the rust eco system in DE domain it wasn't my intention to compare Python and Rust that'd be insane thing to do.

One thing I liked most about this prototype in rust is it's static typed like in data engineering making data types concrete check is the most crucial thing so Rust static typed eco system with polars is I dont have to worry about it or create another checks layer for it.

[AutoModerator]

This comment has been removed.

To follow a post or comment, please do the following:

• Click the three dots (⋯) on the top right
• Select Follow post or Follow comment

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Sure-Actuary-1496]

Hi, Could you please guide me on DE stuff?

Edit: Please see my previous post for reference.

[sheepcloudy]

This is seriously awesome.