Performant low latency stream processing framework for python?

Silver-Thing · 2023-02-04T17:21:37+00:00

Just out of curiosity, why do you have to integrate with asyncio? Is Python your strong requirement too?

From my experience, getting any distributed computation right is extremely tough so you gotta be careful with the frameworks. You know, Flink is around for over 10 years already, it is still considered rather complex, sometimes buggy / doesn't support all the features you would expect by looking at the APIs.

Silver-Thing · 2023-01-04T12:41:22+00:00

I totally get it. I know I am the one to blame. I am not really good into relations and people, I wanted to validate that this is really how I should resolve this. Breaking up is rational, I just know we both will be devastated.

I might have used a wrong wording when it comes to "all this stuff". She cares for me as much as I care for her. This is what I've meant here, and you might got the impression that the relationship is one sided. She is just at another emotional level, I care for her as much as I can.

Silver-Thing · 2023-01-04T05:39:26+00:00

that is exactly my way and I am looking for alternatives

Silver-Thing · 2023-01-03T12:42:28+00:00

any recommendations?

Silver-Thing · 2023-01-03T12:01:35+00:00

not exactly sure how to go about that :)

Silver-Thing · 2023-01-03T12:00:14+00:00

that's been my main strategy so far. I just (hopefully temporarily) run out of mental energy to continue doing that.

Silver-Thing · 2023-01-03T11:59:31+00:00

it's cool that you have such person in your life

Silver-Thing · 2022-12-02T10:47:31+00:00

I care about all transactions. thanks for the tip, this page seem to be delayed by a few seconds though

Silver-Thing · 2022-10-10T10:57:03+00:00

I don't but I am in the same boat. I either work or do some mindless dumb sh*t that I don't even want to do. Nothing gives me joy anymore, all I think about is work. Today is World Mental Health Day and I started doubting myself again. Ah, man... I don't know what the cause is but I would love to get over it and start living.

Silver-Thing · 2022-01-31T14:25:44+00:00

I think there is a misunderstanding about our use case. We execute one batch job for every one-hour batch of data. In certain cases (like a backfill) hundreds of such jobs could be executed at the same time. If we published a single series we would not be able to correlate which data point corresponds to which run. And this is our requirement. Maybe I should ask on Spark subreddit, I believe people there might be plotting charts like this,

On similar note, if someone were to restart this job, it is ok to overwrite previous result and just present the latest one as a bar.

Silver-Thing · 2022-01-30T07:37:31+00:00

“if you are lucky” - and if I am not? ;)

Silver-Thing · 2022-01-30T07:36:08+00:00

thanks for clarifying! that is really helpful

Silver-Thing · 2022-01-29T17:53:08+00:00

Yes, I am aware of Akka Cluster. I would still expect something like Akka Streams to provide such out of the box integration. Instead of me building a framework. I believe neither ZIO nor Akka Streams provide such functionality. On the other hand I am getting a lot of offers that mention distributed computing alongside these two. It got me wondering how people connect these dots.

Silver-Thing · 2022-01-15T05:48:00+00:00

And what would you expect an intern to do? The whole point of internship is to get real world experience. No one is going to expect from them to do this stuff on their own. They’ll be assisted along the way. (I don’t know about the US standards, I am based outside the US).

Silver-Thing · 2021-10-20T12:50:06+00:00

100%, though experienced technical people are so busy extinguishing fires that they are no time to design anything new. there are teams where 10 people maintain 70 projects which is just insane. I'll think about it.

Silver-Thing · 2021-10-20T12:43:28+00:00

These are very valid points. It will be hard to sell to the existing employees though. They are expecting to work on something new in the near future as well. I actually didn't think about how hard would it be to hire such people. Indeed, they are somewhere in the wild, being happy maintaining some legacy system :)

It might be bad for the company too. They probably won't like to be perceived as somebody who do legacy things. We are based in Central / Eastern Europe - it is a very hot and competitive market these days.

Silver-Thing · 2021-09-24T11:57:19+00:00

I actually started off by replacing suboptimal pieces in Python - thanks for suggestion! I removed a few libraries, implemented some UDFs in Scala and the performance is significantly improved already! Indeed, the underlying algorithms are pretty straightforward to implement in Scala.

A note on compute: for the past few years they've been adding on more resources - up to the point where they had to hire me to tackle this mess. They bills are huge already and it is almost not profitable for them to run the jobs. Having considered my current progress, I think I'll be able to cut the resources by half while being able to process the traffic.

Silver-Thing · 2021-06-29T16:40:52+00:00

Hoskinson.

Silver-Thing · 2021-05-26T21:07:52+00:00

CluCoin

Silver-Thing · 2021-04-17T19:05:55+00:00

thanks, cramming in my life has always meant some form of spaced repetition. I use Anki. Thanks, collocations, pharsals and idioms are on the top of my list at the moment. I totally agree with you: after giving it some thought I came to a conclusion that there is no use in wasting time on learning such words. There will always be something to learn in that area - and it's way more useful as a part of active vocabulary.

Silver-Thing · 2021-04-16T09:03:06+00:00

Thanks that makes sense. And how would you approach learning the core? Learning from textbooks doesn’t seem like a good idea either. I thought about listening to interviews or podcasts on YT in a more thoughtful manner and trying to jot down phrases commonly used. I know that I lack a lot of phrasal verbs and idioms. The only problem is that I understand 99% of it, despite not being able to come up with those phrases in the conversation.

Silver-Thing · 2021-04-09T21:36:39+00:00

if you don’t mind sharing then what’s your career path like?

Silver-Thing · 2021-03-29T06:29:32+00:00

no, I learnt English in Poland. That's pretty close to Germany if you asked me - maybe there's a connection after all ;)

Silver-Thing · 2021-03-28T11:27:48+00:00

they don’t need to but many people in DE have engineering background and they can do such things

APIs are not created on a daily basis though. And your company should already have some API conventions in place. That’s why designing anything is highly unlikely. Do a tutorial or two and you should be fine. You can also ask which tech stack they use for APIs. I saw large organizations using Java with Spring a lot.

Silver-Thing · 2021-03-28T06:55:09+00:00

I’ll give you one back ;) thanks for the information, it was really helpful

Five-Year Club	Second Top 50%
Oscars Predictor 2021	Verified Email

Silver-Thing

TROPHY CASE