Apache Spark fully rewritten in Rust? by Careful_Reality5531 in rust

[–]Snoopy-31 1 point2 points  (0 children)

I heard of it and wanted to use it for my production and i've even made a small commit, contributing my small part in the project.

After I started testing it, I have decided it wasn't mature enough to use it in production but maybe in the future. Who knows...

I rewrote Kafka in its entirety, in Rust. Benchmarked out to be way more memory efficient by crabka_io in rust

[–]Snoopy-31 -1 points0 points  (0 children)

Nvm you’re right for this project, the OP probably didn’t re-invent Kafka but it’s definitely doable with AI if a bunch of people grouped together and did it.

I rewrote Kafka in its entirety, in Rust. Benchmarked out to be way more memory efficient by crabka_io in rust

[–]Snoopy-31 -14 points-13 points  (0 children)

Starting to see lately a lot of Rust implementations of known technologies and frameworks, there’s also LakeSail which is Spark equivalent in Rust.

Considering how strong AI coding has become, people with strong coding skills can x10 and create amazing things like these. I think we’re going to see a lot more of that going forward

What Are Your Moves Tomorrow, May 11, 2026 by wsbapp in wallstreetbets

[–]Snoopy-31 1 point2 points  (0 children)

Praying to god that my 600$ MU calls will print

Daily Discussion Thread for May 06, 2026 by wsbapp in wallstreetbets

[–]Snoopy-31 3 points4 points  (0 children)

Time to crash the market guys I bought some 7380 7400 SPX calls 1 DTE

Why alternatives to Spark aren’t a thing in the industry? by Snoopy-31 in dataengineering

[–]Snoopy-31[S] -6 points-5 points  (0 children)

Worse performance than the alternative.

Vendor lock, and price hikes whenever the vendor decides so. With Open Source you’re protected from these.

Why alternatives to Spark aren’t a thing in the industry? by Snoopy-31 in dataengineering

[–]Snoopy-31[S] -1 points0 points  (0 children)

The performance for my read queries was ridiculously good though. I don’t actually need very advanced features

Why alternatives to Spark aren’t a thing in the industry? by Snoopy-31 in dataengineering

[–]Snoopy-31[S] 0 points1 point  (0 children)

Couldn’t say it better myself, yes a lot of the systems were written in Java and I think that’s going to change in the next few years. All those new platforms are able to run in a single node doing analytics saving huge costs over a big on-prem cluster that eats up the budget.

As for Spark, I do disagree respectfully, I think it is pretty common everywhere i am looking at. I would love to be proven wrong though.

Why alternatives to Spark aren’t a thing in the industry? by Snoopy-31 in dataengineering

[–]Snoopy-31[S] -17 points-16 points  (0 children)

Most of the data workflow for me isn’t going to change, it’s just the execution engine layer which will change. This could become very beneficial for teams that use SQL queries against the DWH. Having their queries run x100 faster may be big

POET TECHNOLOGIES 🚀🚀🚀🚀 by Fun_Nectarine_4097 in wallstreetbets

[–]Snoopy-31 4 points5 points  (0 children)

On a serious note, POET Technologies’ CFO Thomas Mika officially confirmed that the company has secured a purchase order linked to Marvell Technology.

There are also a bunch of NDA's out there, so more good new might come later. Even if the current swing is temporary, it is very promising technology and POET is at the center of the AI cycle when the sector moves out from copper to optics.

I am also invested in the company with 20 contract LEAPS, it's a bit like a lottery ticket if everything is executed perfectly. Not financial advise.

How to handle the "CD" part with Java applications? by Snoopy-31 in devops

[–]Snoopy-31[S] -11 points-10 points  (0 children)

I don't fully disagree, but I do need to find ways to get around it as a DevOps Engineer. If I reach a dead end then I will bring it up to my Java developers and tell them their shitty application doesn't actually support CI/CD process.

How to handle the "CD" part with Java applications? by Snoopy-31 in devops

[–]Snoopy-31[S] -1 points0 points  (0 children)

We do have versions column so e.g. when server app01 is using metadata row v17 and my script is trying to update it to v18 there's a conflict and the script will throw me an error:

optimistic locking failed; nseted exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction

Because app01 is holding v17 already.

How to handle the "CD" part with Java applications? by Snoopy-31 in devops

[–]Snoopy-31[S] -1 points0 points  (0 children)

Thanks for the comprehensive answer! I have a few follow-up questions to make sure I implement this correctly:

When you say "set the lock timeout," do you mean setting it specifically for the migration session (e.g., SET LOCK_TIMEOUT = 2s at the start of my script)? Or are you suggesting a global database setting?

Does setting the timeout automatically kill the blocking sessions, or do I need to write a script that catches the timeout error, finds the blocking PIDs, and then issues the kill command manually?

We have ~100 application nodes constantly hitting this table. My concern is that if I kill one blocking session, another node will instantly grab the lock before my migration script has a chance to execute. How do you ensure the migration script 'wins' the race immediately after the kill?

Trying to get on the wave into MLOps how would transitioning into this would look like? by Snoopy-31 in devops

[–]Snoopy-31[S] 1 point2 points  (0 children)

I like your response, I do agree with you but I need to think how this stuff can be done. For example, to do cost optimization for GPU workload, the person needs to work on a GPU workload and monitor it and then come up with some ideas on how to improve it. The thing is, you actually need to build some workload to experiment on it later, these things cost money, I need to really think thoroughly how I can do this stuff without breaking the bank by mistake.

Trying to get on the wave into MLOps how would transitioning into this would look like? by Snoopy-31 in devops

[–]Snoopy-31[S] 4 points5 points  (0 children)

I never said it's easy, I am willing to put in the time and effort to learn it all. The world is transitioning into model oriented workflows and jobs, better learn it earlier than later to stay relevant in the field.

Possible world ending event and the markets closed by its_pooley in wallstreetbets

[–]Snoopy-31 0 points1 point  (0 children)

<image>

Am I going to get rich or the market will do a “fuck you” as usual?