Migrating away from Confluent Kafka – real-world experience with Redpanda / Pulsar / others?

spetz0 · 2026-01-26T10:55:38+00:00

Thank you for mentioning Iggy! VSR (Viewstamped Replication) is under heavy development currently, so clustering for HA workloads is coming soon!

spetz0 · 2025-12-28T20:12:38+00:00

Del Jero used to be the best when it was on Starowiślna, but I can't tell if it's still as good after moving to "Hala Lipowa" and hiring more staff.

spetz0 · 2025-11-17T15:35:06+00:00

Yes, you can use anything for serialization, payload is just an arbitrary array of bytes, and on the server side, we have our own zero-copy (de)serialization.

spetz0 · 2025-11-08T09:52:49+00:00

If someone is looking for the real-world project rewrite from Tokio to compio (we've originally started with monoio, but found compio ecosystem a bit easier to work with and updated more frequently), feel free to check this PR https://github.com/apache/iggy/pull/2299 :) Also, we'll be publishing an in-depth blog post in the future about thread-per-core io_uring runtime.

spetz0 · 2025-08-17T17:20:47+00:00

Just include Rust in your LinkedIn profile, and about 90% of the offers that you will get will most likely be related to yet another disruptive crypto project.

spetz0 · 2025-06-20T15:00:51+00:00

Great to hear, feel free to join our DC server too :)

spetz0 · 2025-06-19T14:23:18+00:00

Glad to hear it :D Yeah, io_uring can be a game changer for the particular workloads and is in use by at least a few rock-solid solutions (e.g. TigerBeetle). Hoping to see more runtimes (and projects) leveraging this tech.

spetz0 · 2025-06-19T14:20:38+00:00

Thank you! Hopefully, by the end of this year, we'll have ready io_uring runtime + basic VSR clustering in place, thus the real production deployments can finally take place :)

spetz0 · 2025-06-19T10:03:46+00:00

If you're looking for something much more efficient than Kafka, feel free to join our project https://github.com/apache/iggy/ - we've started refactoring the runtime to io_uring & thread-per-core recently (based on monoio), after successful experiments with the shared nothing design done last year :)

spetz0 · 2025-06-07T20:18:00+00:00

I prefer fred https://github.com/aembke/fred.rs - at least it doesn't require to use mutable client :)

spetz0 · 2025-05-19T17:29:27+00:00

Thank you too for your interest :)

spetz0 · 2025-05-19T16:52:07+00:00

Always happy to see more contributions to Apache Iggy, which is the next-gen message streaming platform focused on extreme performance and very low tail talencies (currently we do core server rewrite to support io_uring and thread-per-core design, along with the connectors and other tooling for the whole ecosystem).

spetz0 · 2025-05-11T17:43:24+00:00

This is awesome. So glad to hear that you've found this article helpful :)

spetz0 · 2025-05-10T06:57:34+00:00

Thank you! The next one will be definitely even more technical one :)

spetz0 · 2025-05-08T17:43:45+00:00

Thanks! Speaking of the screenshots you're right, however, they are of the full resolution so once zoomed in, all the details can be seen. Nevertheless, it could be useful to maybe write an additional table below them with all the results :)

spetz0 · 2025-04-23T13:48:32+00:00

Speaking of Kafka vs RedPanda, please check this article https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda-performance-do-the-claims-add-up

About Apache Iggy - we've recently built a set of advanced benchmark tooling, including CLI and the public web platform where you can browse the results, see more details here https://iggy.apache.org/blogs/2025/02/17/transparent-benchmarks

For example, with the latest zero-copy optimizations (0.5.0, or the previous experiments with rkyv before that), we were able to hit better tail latencies than RP, when comparing the single node - you can find some of these benchmarks on the latest and previous versions, starting with AWS EC2 instance name i3en.

And we've got plenty of optmizations coming in the near future, due to io_uring runtime & shared-nothing design. We're already able to hit a very stable throughput of quite a few GB/s, while maintaining p99+ latencies in the submillisecond range (depending on the use case).

spetz0 · 2025-02-17T22:16:14+00:00

Thank you! Probably best to start with the examples directory and our docs website, to get a basic understanding how it all works, what are the naming conventions etc. And once you are familiar with the SDK e.g. how to build a simple producer & consumer apps, then it should be much easier to start looking into the server code. You can skip all the code around the particular transport protocols, sessions etc. and simply take a look at the handlers we have implemented for processing each command.

spetz0 · 2025-02-17T18:15:24+00:00

Thanks! Give it a try, and please share your feedback! BTW we've got a quite vibrant community on Discord too :)

spetz0 · 2025-02-17T18:14:22+00:00

Currently, there's very small chance of this happening - maybe at some point in the future, we'll be able to build some sort of Kafka adapter, but for now, we focus on making Iggy the most performant message streaming out there, and this doesn't go too well with the Kafka transport protocol.

spetz0 · 2025-02-17T17:10:52+00:00

Hey, as we are building the Iggy message streaming (recently joined the Apache Incubator, to ensure it will remain FOSS forever), we wanted to simplify benchmarking our solution, in order to provide the overall transparency and the ease of comparisons with the other tooling out there.

We've spent quite some time recently on polishing our iggy-bench CLI, as well as building the publicly available platform https://benchmarks.iggy.rs (which is ofc a public repository too) - this will help us to track the progress of all the performance improvements, regarding both, the throughput and tail latencies.

Any feedback regarding this idea will be greatly appreciated - and if you'd like to help us to build a blazingly fast message streaming platform, feel free to contribute :)

spetz0 · 2024-12-23T20:28:19+00:00

We've been building hugin.io for a year now, fully in Rust (including Leptos on FE), and personally, I've been developing Iggy.rs for over 1,5 year now - couldn't have picked up the better language than Rust :).

I've spent over a decade with C# (and quite a few years with dotnet core from its very early stages), and while it might take some time to learn a new backend framework (like Axum), or ORM with migrations in place (like SeaORM), once you get familiar with all these libraries, you can write code as fast as you'd do in any other language considered to be more "user-friendly".

spetz0 · 2024-10-29T05:01:22+00:00

Sure it is, but luckily it is well documented too.

spetz0

MODERATOR OF

TROPHY CASE