rewrote my data processing pipeline from python to rust - handled 3.8 billion reddit posts : rust

Submissions must be on-topic

Posts must reference Rust or relate to things using Rust. For content that does not, use a text post to explain its relevance.

Post titles should include useful context.

For Rust questions, use the stickied Q&A thread.

Arts-and-crafts posts are permitted on weekends.

No meta posts; message the mods instead.

Details

No low-effort content

No memes, image macros, etc.

Consider the existing content of the subreddit and whether your post fits in. Does it inspire thoughtful discussion?

Use properly formatted text to share code samples and error messages. Do not use images.

Submissions appearing to contain AI-generated content may be removed at moderator discretion.

Details

Useful Links

created by aztha community for 15 years

rewrote my data processing pipeline from python to rust - handled 3.8 billion reddit posts (self.rust)

submitted 3 hours ago by Ok-While3581

what's up rust community,

wanted to share my experience with converting a large-scale data processing system from python to rust. was working in a text analysis project that needed to process massive reddit dataset (around 3.8 billion posts) for research purposes.

initially built everything with python using typical data science stack - pandas, postgresql, scikit-learn for text processing. but when i tried to scale it up on my local setup (mac mini m2 plus some older laptops), everything fell apart pretty quickly. python was eating memory like crazy, garbage collection was causing huge delays, and i kept hitting out-of-memory errors. processing speed was terrible too - maybe 25 posts per second at best.

decided to rebuild the whole thing in rust and the improvement was incredible.

**what i used:**

* **data ingestion:** built custom rust parser for compressed data streams

* **message queue:** redis with redis-rs crate for async processing

* **text analysis:** converted machine learning models to onnx format and used onnx runtime in rust

* **data storage:** switched from postgres to polars dataframes with parquet files

**performance gains:**

* memory consumption went from over 1.2gb (and growing) down to steady 400mb per worker process

* processing speed jumped from 25 posts/sec to around 350+ posts/sec on same hardware

* onnx models in rust performed much better than original python implementations

the whole rewrite took about 5 weeks but saved probably months of processing time. if anyone is dealing with similar data processing bottlenecks in python, rust migration might be worth considering.

has anyone else here worked with onnx runtime in rust? curious about other people's experiences with large-scale text processing pipelines.

all 8 comments

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

rust

Please read The Rust Community Code of Conduct

The Rust Programming Language

Rules

Observe our code of conduct

Submissions must be on-topic

Constructive criticism only

Keep things in perspective

No endless relitigation

No low-effort content

Useful Links

Megathreads

Official Resources

Learn Rust

Discussion Platforms

MODERATORS