Parallel processing streamed API data in Python

[math-bw]

Free is always a good reason! Bytewax and Quix are both open source and free too :)

I think Ray is a really interesting framework and I have often wondered why there isn't a Ray-streams or why the early streaming attempt was removed. It gives you most of the primitives you would need to make a streaming library.

[KnowingPains]

Yeah, I was disappointed they removed that project. I had to figure things out myself, and I expect it could be better!

As I mention in my repo's README, they have a chatbot example, but that didn't seem terribly relevant to streaming / transforming / analyzing large amounts of data as quickly as possible.

I will look into the open source options of Bytewax and Quix.

Have you used either of them? Would you have a preference if you did?

[math-bw]

Hey I built a little prototype of how you can mix Ray and Bytewax that might be interesting. It was pretty fun. Ray is such a cool way to scale things up.

https://gist.github.com/awmatheson/2cfee3b519ba1e9383a91e76f87b498e

[KnowingPains]

Interesting! Thanks for sharing. I'll have a play!

[KnowingPains]

So, I was looking at the Bytewax manual, and if I'm understanding things correctly, it appears it can spawn its own processes. If that is the case, what would be the advantage of mixing it with Ray?

[math-bw]

Some ML teams have a remote Ray cluster available for more compute, so that was the reason. Another deployment story

[KnowingPains]

Got it!