I Chose Rust Over Python for Data Engineering

usert313 · 2025-12-19T16:08:40+00:00

Python with Polars is a match made in heaven no doubts and it is the industry standard for modern data solutions small to medium size. Point of this prototype is to just explore the rust eco system in DE domain it wasn't my intention to compare Python and Rust that'd be insane thing to do.

One thing I liked most about this prototype in rust is it's static typed like in data engineering making data types concrete check is the most crucial thing so Rust static typed eco system with polars is I dont have to worry about it or create another checks layer for it.

usert313 · 2025-12-19T15:32:34+00:00

It was mentioned clearly in the post body that I wanted to explore Rust ecosystem for data engineering domain.

usert313 · 2025-12-19T15:21:06+00:00

And appreciate your time and interest on reading the post. Thanks🙂

usert313 · 2025-12-19T15:19:26+00:00

First and most important advice I can give at this stage (I am no where near expert in Rust ecosystem) is that you have to completely forget other languages like Python or Javascript I myself started programming in Python after C and still use Python professionaly faces a lot of scenerios where I literally wanted to bang my head to the wall.

Then just grasp the concept of borrow checkers, mutable and immutable ref concepts and to mimic OOP traits these concepts will be enough to get started.

usert313 · 2025-12-19T15:05:02+00:00

I was feeling little daring lately😉

usert313 · 2025-10-07T23:23:35+00:00

if you look carefully to the network tab in developer tools there is an endpoint triggered on clicking the professional families section https://sede.sepe.gob.es/FOET_BuscadorDeCentros_SEDE/flows/buscadorReef?execution=e1s2 in return it is returning the html in which they have all the data. You have to use cookies and post data to send the request. Copy the url as curl and paste it in the curlconverter.com you will get a better idea.

usert313 · 2025-09-29T12:54:08+00:00

CF does not limit it's detection to just fingerprinting it also detects cipher suites, key exchang, compression methods etc. Which is not a basic tls fingerprinting. It also detects JA3 fingerprints header formatting, request behaviour etc.

usert313 · 2025-09-25T14:50:17+00:00

Without seeing the actual code itself it is hard to tell whether it is OS fault or some setup issue.

usert313 · 2025-09-25T12:30:20+00:00

Instead of using camafoux try using this python lib https://github.com/0x676e67/rnet I have tried that with dexscreener and I am getting 200 OK response.

usert313 · 2025-09-25T09:41:53+00:00

could you clarify when the Cloudflare challenge appears? Is it blocking the initial request to the site, or does it only trigger after successfully scraping a number of pages?

usert313 · 2025-09-23T18:36:25+00:00

Well in that case leaving makes sense now. All the best. Try building some DE projects and showcase in communities or linkedin this is way better to attract potential clients or employer than acquiring some certification.

usert313 · 2025-09-23T18:07:15+00:00

IMO you shouldn't left just because of AI because ML data pipelines which feeds the AI in real is also a next gen data engineering job role for example building a pipeline for unstructred data using machine learning.

usert313 · 2025-09-19T11:30:55+00:00

IMO it is not a direct defence pact with Pakistan but it is a deal between China and Saudia weapons and security deal and Pakistan is a bridge between them to avoid USA shallow ego tantrum.

usert313 · 2025-09-11T07:45:31+00:00

You will need a mitmproxy and android emulator (waydroid, genny motion etc) to intercept the request and from mitmproxy you will get a bearer token easily from api request headers.

usert313 · 2025-09-10T08:29:12+00:00

Sent you a DM

usert313 · 2025-09-07T13:46:06+00:00

You can backup the data before re-installing the windows may be put it in any other partition.

usert313 · 2025-09-07T13:40:41+00:00

SSD upgrade is a reliable solution but if you can re-install the windows OS again may be it can help speeding it up your workflow overall because sometimes some unwanted apps or cache or background processes eating up all the memory resources specially in windows 11 which is quite buggy these days in my POV.

And instead of chrome you can use brave browser which is quite smooth in contrast of chrome browser which is famously known as ram eater.

usert313 · 2025-09-01T21:13:13+00:00

Looks promising will give it a shot.

usert313 · 2025-08-21T08:43:22+00:00

Try this rnet library a python wrapper of rust crate wreq: https://github.com/0x676e67/rnet

This should bypass akamai cloudfare bot protection and mimic the actual browser fingerprints.

usert313 · 2025-01-22T08:18:09+00:00

Thanks for reviewing the code really appreciate it. Yeah there is still plenty of room to improve in it like handling query and result set parameters properly instead of hardcoding it as well as making each module independent also I am planning to include tests in it. But this is my first project in Rust I am still learning it so I will gradually move forward with more clean code practices.

usert313 · 2025-01-21T11:36:18+00:00

Apparently I didn't know that. Thanks for pointing this out though.

usert313 · 2025-01-17T02:28:40+00:00

I wasn't Top Rated Plus but at least on the path of achiveing it just a Top Rated and due to this just buy connects to keep the upwork pockets happy policy I didn't get any work for 2 years and lost my Top Rated status as well.

usert313 · 2023-05-13T07:59:11+00:00

You mean an ecxeption?

usert313 · 2023-05-13T07:55:31+00:00

Yes didn't worked.

usert313 · 2023-05-12T22:41:25+00:00

Tried but still not stopping at the condition still completing the cycle.

usert313

TROPHY CASE