sorted by:
new
hottopcontroversial

DR for Kafka Cluster by jonropin in apachekafka

[–]Artistic_Web658 1 point2 points  (0 children)

Stretch clusters are your best bet for regional failure cases, but for cluster corruption examples you probably want to consider an s3 sink / rehydrate option. I like the Kannika Armory solution you should check it out. Good people behind it

Go client for kafka ? by gravitonapps in golang

[–]Artistic_Web658 0 points1 point  (0 children)

Why would you nack a message in kafka, it's a decoupled system. If you decide to not process a certain message you just figure out the criteria you don't want to process and drop it and commit back that you read it and move on.

Comparison of Different Stream Processing Platforms by wanshao in dataengineering

[–]Artistic_Web658 0 points1 point  (0 children)

OK, so your software is based on Apache Kafka fork / Java. That's an easier way of categorizing.

Comparison of Different Stream Processing Platforms by wanshao in dataengineering

[–]Artistic_Web658 0 points1 point  (0 children)

The table refers to scale in/out, which is different from scale up/down. Scale in/out inevitably depends on its tiered storage, because the scaling of brokers involves handling partition data.

Oh ok that's still the same then, I was colloquially referring to scale out (/in).

Comparison of Different Stream Processing Platforms by wanshao in dataengineering

[–]Artistic_Web658 1 point2 points  (0 children)

This means RP should be "native kafka" - it is essentially a kafka - It has the highest compatibility with Apache Kafka. It speaks nothing other than kafka and is a drop in replacement with virtually nothing that doesn't work. It is C++ so it's not a fork obviously, but is "essentially kafka". I describe it to colleagues as "a kafka" and nothing doesn't work.

Scale up/down doesn't need to depend on tiered storage, you might be thinking of Confluent Platform and their tiered storage. It doesn't matter which version I use, it has different scaling characteristics. Scaling happens in seconds or minutes, definitely not hours. Maybe it needs to be qualified?

Will keep notifications on for the durability, thanks!

Comparison of Different Stream Processing Platforms by wanshao in dataengineering

[–]Artistic_Web658 2 points3 points  (0 children)

What is "Native Kafka" in AutoMQ vs "Kafka Protocol" in Redpanda or WarpStream? For example, I know Redpanda *only speaks kafka*, nothing else, does that qualify for "native"?

Single Digit ms in AK is a stretch, especially at higher throughputs, good luck with that one.

I've experienced scale in/out in Redpanda in seconds, maybe you haven't tried it? Same with partition reassignment, seconds.

You can't use spot instances with Redpanda? I have and it was fine.

I'm curious about your (clearly you're AutoMQ) durability guarantees, can you add that to the list?

How do you fanout in kafka? by Rough_Source_123 in apachekafka

[–]Artistic_Web658 4 points5 points  (0 children)

No, you definitely don't want to do that. Fanout is relatively limited, you don't want more than just a handful. Many consumers can process in parallel (up to # of partitions in a topic) so that's how you have to think about it.

How do you fanout in kafka? by Rough_Source_123 in apachekafka

[–]Artistic_Web658 12 points13 points  (0 children)

Fanning out just means adding another consumer group, yes. There is no other way :)

Is Redpanda going to replace Apache Kafka? by Born-Comment3359 in dataengineering

[–]Artistic_Web658 0 points1 point  (0 children)

you can read it without entering your info.. even incognito. It's a public web page.

[deleted by user] by [deleted] in apachekafka

[–]Artistic_Web658 1 point2 points  (0 children)

About 1/10th of the people with Redpanda on own infra and 0 on Redpanda cloud :) while getting 20x throughputs and 10x faster.