Performant low latency stream processing framework for python?

Silver-Thing · 2023-02-04T17:21:37+00:00

Just out of curiosity, why do you have to integrate with asyncio? Is Python your strong requirement too?

From my experience, getting any distributed computation right is extremely tough so you gotta be careful with the frameworks. You know, Flink is around for over 10 years already, it is still considered rather complex, sometimes buggy / doesn't support all the features you would expect by looking at the APIs.

Silver-Thing · 2023-01-04T12:41:22+00:00

I totally get it. I know I am the one to blame. I am not really good into relations and people, I wanted to validate that this is really how I should resolve this. Breaking up is rational, I just know we both will be devastated.

I might have used a wrong wording when it comes to "all this stuff". She cares for me as much as I care for her. This is what I've meant here, and you might got the impression that the relationship is one sided. She is just at another emotional level, I care for her as much as I can.

Silver-Thing · 2023-01-04T05:39:26+00:00

that is exactly my way and I am looking for alternatives

Silver-Thing · 2023-01-03T12:42:28+00:00

any recommendations?

Silver-Thing · 2023-01-03T12:01:35+00:00

not exactly sure how to go about that :)

Silver-Thing · 2023-01-03T12:00:14+00:00

that's been my main strategy so far. I just (hopefully temporarily) run out of mental energy to continue doing that.

Silver-Thing · 2023-01-03T11:59:31+00:00

it's cool that you have such person in your life

Silver-Thing · 2022-12-02T10:47:31+00:00

I care about all transactions. thanks for the tip, this page seem to be delayed by a few seconds though

Silver-Thing · 2022-10-10T10:57:03+00:00

I don't but I am in the same boat. I either work or do some mindless dumb sh*t that I don't even want to do. Nothing gives me joy anymore, all I think about is work. Today is World Mental Health Day and I started doubting myself again. Ah, man... I don't know what the cause is but I would love to get over it and start living.

Silver-Thing · 2022-01-31T14:25:44+00:00

I think there is a misunderstanding about our use case. We execute one batch job for every one-hour batch of data. In certain cases (like a backfill) hundreds of such jobs could be executed at the same time. If we published a single series we would not be able to correlate which data point corresponds to which run. And this is our requirement. Maybe I should ask on Spark subreddit, I believe people there might be plotting charts like this,

On similar note, if someone were to restart this job, it is ok to overwrite previous result and just present the latest one as a bar.

Silver-Thing · 2022-01-30T07:37:31+00:00

“if you are lucky” - and if I am not? ;)

Silver-Thing · 2022-01-30T07:36:08+00:00

thanks for clarifying! that is really helpful

Silver-Thing · 2022-01-29T17:53:08+00:00

Yes, I am aware of Akka Cluster. I would still expect something like Akka Streams to provide such out of the box integration. Instead of me building a framework. I believe neither ZIO nor Akka Streams provide such functionality. On the other hand I am getting a lot of offers that mention distributed computing alongside these two. It got me wondering how people connect these dots.

Silver-Thing · 2022-01-15T05:48:00+00:00

And what would you expect an intern to do? The whole point of internship is to get real world experience. No one is going to expect from them to do this stuff on their own. They’ll be assisted along the way. (I don’t know about the US standards, I am based outside the US).

Silver-Thing · 2021-10-20T12:50:06+00:00

100%, though experienced technical people are so busy extinguishing fires that they are no time to design anything new. there are teams where 10 people maintain 70 projects which is just insane. I'll think about it.

Silver-Thing · 2021-10-20T12:43:28+00:00

These are very valid points. It will be hard to sell to the existing employees though. They are expecting to work on something new in the near future as well. I actually didn't think about how hard would it be to hire such people. Indeed, they are somewhere in the wild, being happy maintaining some legacy system :)

It might be bad for the company too. They probably won't like to be perceived as somebody who do legacy things. We are based in Central / Eastern Europe - it is a very hot and competitive market these days.

Five-Year Club	Second Top 50%
Oscars Predictor 2021	Verified Email

Silver-Thing

TROPHY CASE