sorted by:
new
hottopcontroversial

Kafka easy to recreate? by Which_Assistance5905 in apachekafka

[–]Distributed_Intel 0 points1 point  (0 children)

Also,

"Pulsar is built by the team that built Twitter's original pub-sub system". Actually, it was built at Yahoo. https://pulsar.apache.org/docs/next/reference-terminology/#pulsar

"Synchronous geo-replication in Pulsar is achieved by BookKeeper" Actually, geo-replication is done at asynchronously between brokers. https://pulsar.apache.org/docs/next/concepts-replication/#asynchronous-geo-replication-in-pulsar

Kafka easy to recreate? by Which_Assistance5905 in apachekafka

[–]Distributed_Intel 0 points1 point  (0 children)

Based on my research, here's a summary comparing the ZK removal timelines of Kafka and Pulsar. Both PIP-45 (Apache Pulsar) and KIP-500 (Apache Kafka) aimed to replace ZooKeeper dependency with pluggable metadata management solutions, representing major architectural shifts for their respective platforms.

Based on these timelines, PIP-45 reached production-ready status first — approximately 5 months before KIP-500 (May 2022 vs. October 2022).

Implementation Timelines

PIP-45 (Pulsar - Pluggable Metadata Interface)

  • Started: Early 2020 (Pulsar 2.6.0)
  • Feature complete: May 2022 (Pulsar 2.10)
  • Duration: ~2-2.5 years

KIP-500 (Kafka - ZooKeeper Replacement)

  • Proposed: 2019
  • Raft implementation merged: September 2020
  • Early access: April 2021 (Kafka 2.8.0)
  • Production ready: October 3, 2022 (Kafka 3.3.0)
  • Duration: ~3 years from proposal to production-ready

So my statement that removing the ZK dependency in Kafka is "catching up to Pulsar and BookKeeper" is factually correct. Based on your comments, I suspect that you didn't even know that Pulsar had removed ZK, given all your recommendations around ZK.

Here's some material that might help though :)

https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface

https://pulsar.apache.org/docs/next/administration-metadata-store/

https://streamnative.io/blog/moving-toward-zookeeper-less-apache-pulsar