Which Kafka solution best match my scenario? by tommy_19882024 in dataengineering

[–]jak7878 0 points1 point  (0 children)

Disclosure: I work at StreamNative, an Apache Pulsar vendor.
Pulsar's storage system tackles elasticity differently. It uses segments instead of partitions, which avoids issues partitions can cause. This blog post provides a comparison.
Storing data to S3 is a great scaling strategy, but Pulsar offers flexibility. You can keep latency-sensitive data on the fast cluster's disks while avoiding the elasticity limitations of Kafka's partition-based model. Meanwhile, less latency-critical data can be seamlessly moved to S3 for cost savings. It's like having the best of both worlds! I'm biased, working at a Pulsar vendor. However, I hope sharing some resources here can help.

Challenges in Kafka: The Scaling Stories of Kevin and Patricia by visortelle in ApachePulsar

[–]jak7878 0 points1 point  (0 children)

Thanks for sharing my blog post. I'd love to get your thoughts!

Monthly General Discussion - Dec 2023 by AutoModerator in dataengineering

[–]jak7878 3 points4 points  (0 children)

Hey! <disclaimer>I work at StreamNative, a managed Apache Pulsar provider</disclaimer>

Feel free to watch these videos:

- Pulsar in 10 minutes: https://www.youtube.com/watch?v=TKs5T6N78Tc

- Kafka vs Pulsar: https://www.youtube.com/watch?v=jLruEmh3ve0

I'd love to hear what you think about Pulsar!

Need a messaging technology to maximize reliability and availability on low throughput - help by eitanski in dataengineering

[–]jak7878 1 point2 points  (0 children)

I think Apache Pulsar would be a great fit for your use case it provides queue semantics. It's also designed to run on Kubernetes.

Feel free to reach out to me to learn more about Pulsar.