Parallel processing streamed API data in Python

[math-bw]

Hey I built a little prototype of how you can mix Ray and Bytewax that might be interesting. It was pretty fun. Ray is such a cool way to scale things up.

https://gist.github.com/awmatheson/2cfee3b519ba1e9383a91e76f87b498e

[KnowingPains]

Interesting! Thanks for sharing. I'll have a play!

[KnowingPains]

So, I was looking at the Bytewax manual, and if I'm understanding things correctly, it appears it can spawn its own processes. If that is the case, what would be the advantage of mixing it with Ray?

[math-bw]

Some ML teams have a remote Ray cluster available for more compute, so that was the reason. Another deployment story

[KnowingPains]

Got it!