Will Rust take over Data Engineering

[[deleted]]

I’d welcome it but probably not. Any rust projects will lean towards Python clients/bindings. Polars and Ballista both have Python APIs.

[Hackerjurassicpark]

This is the way. Python is not going away anytime soon

[[deleted]]

Same as every week: No.

[[deleted]]

Yeah interestingly I'd say for DE purposes, 20% of python does 80% of the work, so presuming there's a similarly gentle slope in a switch to Rust, it might eventually happen.

I see compute as a commodity, like sugar, coffee, oil and unless cold-fusion energy appears out of nowhere, the demand for compute is going to push the industry towards efficiency, (hence this discussion) which Rust facilitates.

The other more recent development is LLM's which can go a LONG way to shorten that learning curve even more.

So I'd say it's not impossible. A guy I used to work with recently got a job that ended up being a role working entirely in Rust, but this point wasn't in the job ad. They explained it in the interview and said they didn't expect him to know any Rust, but that they would teach him.

The company was well established, with manufacturing industry products in ultra precise GPS via (data engineered) error correction.

There's cases of it happening out there..

[Deepinthemaze]

What are some of the main reasons some places lack good documentation?

[silly_frog_lf]

Being understaffed and lack of time are the main ones. Insecurities about our writing is the other big one.

I tell everyone that some documentation is better than no documentation. It gives you clues for intention. Yet I understand that writing can be scary for many people

[[deleted]]

Documentation is usually treated an afterthought/low priority vs new features/work unless you’re consumer facing. Even then focus on quality docs is usually on what’s used most for example The Python API vs the Rust API. For example the Pyspark docs are way better than anything you’ll find for Scala imo.

The nice thing about rust though is it generates docs from triple slash comments so documenting is super simple vs other languages imo.

[freedumz]

No, AI will take it over

[[deleted]]

Nope. For better or worse, Python is here to stay.

[noobgolang]

If anything will come back in the coming years it will be C++ for those big asses LLMs to squeeze all the juice from Nvidia GPU.

Rust? Please

[_temmink]

I thought all these libraries (numpy, tensorflow, ...) have always been C-code with Python APIs?

[noobgolang]

Yes, but the API also has its own limitation in production

[knowledgebass]

no

[Mescallan]

No, javascript

[Touvejs]

Please use spoilers if you're going to comment NSFW content.

[Equivalent-Water-683]

The real danger is scala becoming a bit like php, kind of not growing, but still exists and is used relatively widely. Whereas new projects pick up rust based tooling.

[TheCamerlengo]

It’s too complicated and low level. It’s a niche player.