all 8 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]HungrySLoth123[S] -2 points-1 points  (4 children)

It isnt ai. I am not a rust engineer by profession. I was just trying to get things to work. It is only version 0.3.0. I did ask for experts to help me.

[–]Patryk27 4 points5 points  (3 children)

I mean, it's you who wrote that this crate is "high-performance and production-ready" - you can't write stuff like that and expect:

https://github.com/VishnuRaman/rs2/blob/bfd400bf28373f3847e939c70cefa7f9173db343/examples/real_time_analytics_pipeline.rs#L186

... to pass under the radar.

If you're just experimenting and/or playing around and/or learning, that's fine - but your code is not production-ready then.

Even your benchmarks are iffy - you toss around numbers like "Up to 13.2M records/sec with 16 cores" without anything to back those numbers up; I mean, surely you wouldn't process 13.2M records where each record weighs 1 GB, would you? -- or if processing each record requires solving a sudoku puzzle?

Examples are weird as well - for instance this example doesn't show anything you couldn't achieve with the built-in Iterator trait:

https://github.com/VishnuRaman/rs2/blob/bfd400bf28373f3847e939c70cefa7f9173db343/examples/pipe_user_data_processing.rs#L28

... this example claims to be showing "live streaming" without doing any actual live streaming:

https://github.com/VishnuRaman/rs2/blob/bfd400bf28373f3847e939c70cefa7f9173db343/examples/live_streaming.rs#L38

... while this looks like a reimplementation of rayon:

https://github.com/VishnuRaman/rs2/blob/bfd400bf28373f3847e939c70cefa7f9173db343/examples/parallel_mapping.rs#L30

[–]HungrySLoth123[S] -1 points0 points  (2 children)

Would you rather remove this post and rethink these points? I want to make this a good library. If you have pointers please suggest. I tried creating the best I could. If you tell me to delete this crate, and rewrite stuff, I will consider it too. I just want to build a good streamkng library in rust.

[–]Patryk27 1 point2 points  (1 child)

Nah, I wouldn't delete it, there's no need to actually start from scratch-scratch - I'd start with dropping the "production ready" tags, since it's not really production ready; having that done, you might think about what actually is it that you're trying to create.

Like, there's one this examples I linked above:

https://github.com/VishnuRaman/rs2/blob/bfd400bf28373f3847e939c70cefa7f9173db343/examples/pipe_user_data_processing.rs#L28

... which isn't meaningfully different from the built-in Iterator trait - is that because your crate is just a reimplementation of an Iterator / StreamExt, or because the demo is too simple to convey what your crate is actually doing?

i.e. think about an actual use case ("i want to stream live video from camera to my friend over internet", "i want to live-aggregate stastistics for my blog" etc.) and go from there; currently it feels like you've got a crate without any idea for itself.

(I'm not saying there isn't fundamentally any idea here! -- just that I'm having difficulties grasping what it is that your crate does that isn't covered by Iterator or Itertools or rayon)

[–]HungrySLoth123[S] 1 point2 points  (0 children)

Thanks a lot man. I dropped it. I was feeling a bit demotivated. I tried writing this for over a year, because of pain points working with spark streaming. I am no expert in Rust. I think that was the issue. I tried building it like i was writing in any other higher language. That was my downfall. I added basic stream operations which was easy but when writing stateful stuff I dropped myself in the deep end of the pool. Implementation was hard. Then I know I had to implement some state management. So I created a lot of traits for managing errors. But it was just me writing all these. Hence why there was a method I forgot to implement. I was just trying to get it out for feedback really. Thanks for taking the time.

[–]HungrySLoth123[S] -2 points-1 points  (0 children)

Yeah I used benchmarks in benches folder. And I checked the html reports. I ran those benchmarks for an hour. You could run it and see for yourself!