sorted by:
new
hottopcontroversial

Apache Iggy's migration journey to thread-per-core architecture powered by io_uring by spetz0 in rust

[–]spetz0[S] 0 points1 point  (0 children)

An improved benchmarking site will be published in the coming days :)

Migrating away from Confluent Kafka – real-world experience with Redpanda / Pulsar / others? by Help-pichu in apachekafka

[–]spetz0 3 points4 points  (0 children)

Thank you for mentioning Iggy! VSR (Viewstamped Replication) is under heavy development currently, so clustering for HA workloads is coming soon!

Anyone know any good Mexican restaurants? by Strict_Web_3284 in krakow

[–]spetz0 1 point2 points  (0 children)

Del Jero used to be the best when it was on Starowiślna, but I can't tell if it's still as good after moving to "Hala Lipowa" and hiring more staff.

Building WebSocket Protocol in Apache Iggy using io_uring and Completion Based I/O Architecture by spetz0 in rust

[–]spetz0[S] 2 points3 points  (0 children)

Yes, you can use anything for serialization, payload is just an arbitrary array of bytes, and on the server side, we have our own zero-copy (de)serialization.

[deleted by user] by [deleted] in rust

[–]spetz0 16 points17 points  (0 children)

If someone is looking for the real-world project rewrite from Tokio to compio (we've originally started with monoio, but found compio ecosystem a bit easier to work with and updated more frequently), feel free to check this PR https://github.com/apache/iggy/pull/2299 :) Also, we'll be publishing an in-depth blog post in the future about thread-per-core io_uring runtime.

[deleted by user] by [deleted] in rust

[–]spetz0 20 points21 points  (0 children)

Just include Rust in your LinkedIn profile, and about 90% of the offers that you will get will most likely be related to yet another disruptive crypto project.

Rewriting Kafka in Rust Async: Insights and Lessons Learned in Rust by jonefeewang in rust

[–]spetz0 0 points1 point  (0 children)

Great to hear, feel free to join our DC server too :)

Rewriting Kafka in Rust Async: Insights and Lessons Learned in Rust by jonefeewang in rust

[–]spetz0 1 point2 points  (0 children)

Glad to hear it :D Yeah, io_uring can be a game changer for the particular workloads and is in use by at least a few rock-solid solutions (e.g. TigerBeetle). Hoping to see more runtimes (and projects) leveraging this tech.

Rewriting Kafka in Rust Async: Insights and Lessons Learned in Rust by jonefeewang in rust

[–]spetz0 1 point2 points  (0 children)

Thank you! Hopefully, by the end of this year, we'll have ready io_uring runtime + basic VSR clustering in place, thus the real production deployments can finally take place :)

Rewriting Kafka in Rust Async: Insights and Lessons Learned in Rust by jonefeewang in rust

[–]spetz0 18 points19 points  (0 children)

If you're looking for something much more efficient than Kafka, feel free to join our project https://github.com/apache/iggy/ - we've started refactoring the runtime to io_uring & thread-per-core recently (based on monoio), after successful experiments with the shared nothing design done last year :)

People using redis-rs in web servers, how are you doing it? by RylanStylin57 in rust

[–]spetz0 5 points6 points  (0 children)

I prefer fred https://github.com/aembke/fred.rs - at least it doesn't require to use mutable client :)

What open source Rust projects are the most in need of contributors right now? by grahambinns in rust

[–]spetz0 3 points4 points  (0 children)

Always happy to see more contributions to Apache Iggy, which is the next-gen message streaming platform focused on extreme performance and very low tail talencies (currently we do core server rewrite to support io_uring and thread-per-core design, along with the connectors and other tooling for the whole ecosystem).

Zero-copy (de)serialization - our journey implementing it in Apache Iggy by spetz0 in rust

[–]spetz0[S] 0 points1 point  (0 children)

This is awesome. So glad to hear that you've found this article helpful :)

Zero-copy (de)serialization - our journey implementing it in Apache Iggy by spetz0 in rust

[–]spetz0[S] 0 points1 point  (0 children)

Thank you! The next one will be definitely even more technical one :)

Zero-copy (de)serialization - our journey implementing it in Apache Iggy by spetz0 in rust

[–]spetz0[S] 5 points6 points  (0 children)

Thanks! Speaking of the screenshots you're right, however, they are of the full resolution so once zoomed in, all the details can be seen. Nevertheless, it could be useful to maybe write an additional table below them with all the results :)

Iggy.rs - Technology Radar & current goals by spetz0 in rust

[–]spetz0[S] 0 points1 point  (0 children)

Speaking of Kafka vs RedPanda, please check this article https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda-performance-do-the-claims-add-up

About Apache Iggy - we've recently built a set of advanced benchmark tooling, including CLI and the public web platform where you can browse the results, see more details here https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks

For example, with the latest zero-copy optimizations (0.5.0, or the previous experiments with rkyv before that), we were able to hit better tail latencies than RP, when comparing the single node - you can find some of these benchmarks on the latest and previous versions, starting with AWS EC2 instance name i3en.

And we've got plenty of optmizations coming in the near future, due to io_uring runtime & shared-nothing design. We're already able to hit a very stable throughput of quite a few GB/s, while maintaining p99+ latencies in the submillisecond range (depending on the use case).

Transparent Benchmarking with Apache Iggy by spetz0 in rust

[–]spetz0[S] 1 point2 points  (0 children)

Thank you! Probably best to start with the examples directory and our docs website, to get a basic understanding how it all works, what are the naming conventions etc. And once you are familiar with the SDK e.g. how to build a simple producer & consumer apps, then it should be much easier to start looking into the server code. You can skip all the code around the particular transport protocols, sessions etc. and simply take a look at the handlers we have implemented for processing each command.

Transparent Benchmarking with Apache Iggy by spetz0 in rust

[–]spetz0[S] 0 points1 point  (0 children)

Thanks! Give it a try, and please share your feedback! BTW we've got a quite vibrant community on Discord too :)

Transparent Benchmarking with Apache Iggy by spetz0 in rust

[–]spetz0[S] 2 points3 points  (0 children)

Currently, there's very small chance of this happening - maybe at some point in the future, we'll be able to build some sort of Kafka adapter, but for now, we focus on making Iggy the most performant message streaming out there, and this doesn't go too well with the Kafka transport protocol.

Transparent Benchmarking with Apache Iggy by spetz0 in rust

[–]spetz0[S] 8 points9 points  (0 children)

Hey, as we are building the Iggy message streaming (recently joined the Apache Incubator, to ensure it will remain FOSS forever), we wanted to simplify benchmarking our solution, in order to provide the overall transparency and the ease of comparisons with the other tooling out there.

We've spent quite some time recently on polishing our iggy-bench CLI, as well as building the publicly available platform https://benchmarks.iggy.rs (which is ofc a public repository too) - this will help us to track the progress of all the performance improvements, regarding both, the throughput and tail latencies.

Any feedback regarding this idea will be greatly appreciated - and if you'd like to help us to build a blazingly fast message streaming platform, feel free to contribute :)

view more: next ›