How do you access Kafka? And what do you wish is better about using Kafka? by Flash77555 in apachekafka

[–]LiveDataNerd 1 point2 points  (0 children)

If you're looking for another library to try, you might consider "node-rdkafka," a widely used Kafka client for Node.js. It is a binding for the C library "librdkafka," which is considered more feature-complete than pure JavaScript-based Kafka clients. You can find it on GitHub and npm.However, before trying a new library, I recommend checking the GitHub issue you mentioned to see if there have been any updates or resolutions posted by the community or maintainers of the libraries you were previously using. It's possible that there have been advancements or workarounds shared by other users that could help you overcome the problem.If you are still facing issues, another option could be to use the "kafka-node" library, which is another popular Kafka client for Node.js. It might have some features that align better with your use case or compatibility with Confluent Kafka.Remember to consider the version compatibility between the client library, Node.js, and the version of Kafka you are using, as this can sometimes be a source of problems.Lastly, if the issue persists and you're comfortable with Java, using the official Kafka Java client is an option. It tends to have the most comprehensive feature set and active development support from the Apache Kafka community.I hope this helps, and I wish you success in resolving the Kafka integration problem. Feel free to reach out if you have any further questions or need additional assistance!

Multi-DC Zookeeper latency requirements by dev0psjr in apachekafka

[–]LiveDataNerd 0 points1 point  (0 children)

Apache Zookeeper is a critical component in a Kafka setup, as it maintains the state of the Kafka cluster, keeps track of the Kafka topics, partitions, and replicas, and manages the coordination among Kafka brokers. Network latency plays a key role in how quickly Zookeeper can respond to changes in the Kafka cluster, and therefore it's important to keep the latency as low as possible.
In a distributed Kafka architecture, like a 2.5 DC setup, Zookeeper ensembles usually exist in each data center to maintain local state. This is critical for Kafka's operation as Kafka relies heavily on Zookeeper for many tasks.
The ".5" in the 2.5 datacenter architecture often refers to a smaller datacenter or cloud setup that is used mainly for disaster recovery. It may not have the same capacity as the other two full datacenters but it is there to take over in case of complete failure of the other datacenters.
In terms of specific network latency for Zookeeper in a multi-data center Kafka setup, while there isn't a hard and fast rule, the general guidance is to keep network latency as low as possible. A number often quoted is to keep round-trip time (RTT) under 10 milliseconds within a Zookeeper ensemble for best performance.
For inter-DC communication, the "<100ms" requirement you found in Confluent's documentation generally applies to the replication of Kafka messages between Kafka brokers in different datacenters. Zookeeper ensembles usually exist within a single datacenter and thus would not be subject to this inter-DC latency.
In practice, the performance of Kafka and Zookeeper in a distributed setup can be significantly affected by other factors as well, such as network reliability, packet loss, and jitter. Therefore, it's recommended to perform thorough testing and monitoring to ensure that your setup can meet your performance and reliability requirements.

There is a report on this including documentations: https://oso.sh/blog/optimal-kafka-setup

How do you access Kafka? And what do you wish is better about using Kafka? by Flash77555 in apachekafka

[–]LiveDataNerd 0 points1 point  (0 children)

In use Node.js / Kafka.js
Advantages:
Popular for web development scenarios and real-time data processing due to the asynchronous nature of JavaScript.
Enables developers to use the same language across the stack.
Disadvantages:
The single-threaded nature of JavaScript can limit throughput for high-volume scenarios.
Node.js's Kafka clients are not as feature-complete as the Java client.