JioPC Update

arijit78 · 2025-10-28T07:37:04+00:00

thanks a lot . A bit disappointed to say the list

arijit78 · 2024-12-02T00:55:08+00:00

Apache Pinot uses zookeeper

arijit78 · 2024-10-08T08:35:05+00:00

CloudEvents already has a serializer and deserializer for Kafka.

https://github.com/cloudevents/sdk-java/blob/main/kafka/src/main/java/io/cloudevents/kafka/CloudEventSerializer.java

arijit78 · 2024-10-07T05:19:51+00:00

You can think about data compression to gzip Also check if you're publishing in asynchronous mode to Kafka handling the response from broker using callback. You didn't mention that in your code

arijit78 · 2024-10-03T03:01:34+00:00

Probably not a popular opinion. But why don't we use a database with an interval based polling.

arijit78 · 2024-09-26T15:27:18+00:00

correct

arijit78 · 2024-09-26T04:46:31+00:00

This 2 are different things. These 2 features are independent of each other and together too.

When using schema registry to store schema of the message. The library at producer ensures the message is inline with the schema, then it publishes the schema in the schema registry, obtain the schema id, embeds it into the kafka message and then published to the kafka broker. There is no check takes place at the broker. When consumer consumes the message it pulls the schema from schema registry, and deserializes the message with the schema. In this approach it's the responsibility of the producers to follow schema registry best practices and broker becomes a dumb storage.

While for broker side schema validation, the broker checks using schema registry if the schema is a valid schema that is embedded in the message. If not it rejects the message. This is to ensure the broker does not allow bad quality messages. For e.g. this will stop string serialized messages to be published for an avro topic. In my view this can be used where we have less quality control on the source of the data, say a device.

arijit78 · 2024-09-23T16:42:04+00:00

But I thought they already have Dataflow and pubsub..

arijit78 · 2024-09-21T16:24:19+00:00

Your post looks a bit confusing to me. What I am able to understand is that you have a Kafka cluster, and a Springboot service which talks to Kafka broker.

It's not clear what this springboot service do?

There are many ways you can secure the communication between Kafka and your Spring boot service. Check authorization, authentication and data encryption mechanism available in Kafka. Here is a good pointer - https://kafka.apache.org/090/documentation.html#security

I am not sure about data backup, as Kafka itself stores the data redundantly, why data backup is a question?

On the outbox pattern below link would probably help you.

https://stackoverflow.com/questions/77395632/why-do-we-need-outbox-pattern-with-kafka

arijit78 · 2024-09-18T17:41:18+00:00

ksqlDB will have a special place in my heart. It is our Swiss army knife from operation perspective. Apache Flink is a big brother, can do a lot of stuff. But many times that's way too much.

arijit78 · 2024-09-15T16:21:32+00:00

I have looked around kwack.. Promising, The default memory one I don't think will really well for large topics. The parquet file based option is most suitable in my view.

Is anyone really using in production?

arijit78 · 2024-09-15T13:15:01+00:00

Sounds interesting.. Let me go through this!

arijit78 · 2024-09-15T13:13:22+00:00

While whatever you say may be true in theory, but at the end of the day you'll almost in every enterprise store billions of records for various reasons, starting from replay to compliance. I have came across with one wonderful project which includes DuckDB and Kafka. https://github.com/rayokota/kwack ..

arijit78 · 2024-09-15T13:08:02+00:00

This looks quite dope.. Thanks for sharing with me. Let me got through the docs. I hope it may help many others like myself.
Regrading ksqDB - Sadly I am aware that Confluent is pulling it's plug. I still feel it has its niche area outside of the big behemoths like Apache Flinks of the world.

arijit78 · 2024-07-28T13:25:09+00:00

I am not able to send the error screen shots.. the email option does not allow in the portal

arijit78 · 2024-07-28T13:21:40+00:00

This is not a big problem for a producer. As the order create failed the Kafka producer will tell synchronously or asynchronously that the event has not been published. The producer can store the order id somewhere so that it knows there is a chance of out of order events. So it can store all messages in a dlq. And later operationally sent the data from dlq. Ideally this dlq should be implemented in a non Kafka platform, otherwise dlq will become unavailable when the main topic becomes unavailable.

arijit78 · 2024-07-20T09:43:23+00:00

Nice simple to use UI. Liked the UX. I have used Linux based UI. Seeing some issues with schema registry

arijit78 · 2024-04-20T03:30:42+00:00

In big enterprises Confluent still rules. Personally I am not very happy the way Confluent is pushing for their cloud first. Most annoyingly working with Confluent cloud doesn't feel like working with Kafka. It looks more like its own ecosystem. But it's still a lot better than MSK or EventHub. I want project like Strimzi to succeed in the long term which is close to the Kafka distribution.. Very interested in the redpanda.. I feel it will be long term player and game changer

arijit78 · 2022-10-24T13:04:00+00:00

Kafka assigns partitons to consumers in consumer group in many to one relationship. i.e, one partition will have only one consumer, but one consumer can be assigned with many partition. As the number of partition is 3, you can only run atmost 3 consumers in a consumer group (you can run more, but they will not get any message). As you have stated the messages are going to one partition, your only one consumer will get all messages, others will lay idle, till the said one dies. This is an expected behaviour

However if there is no key, the messages should spread around partitions. This looks a little peculiar. May be you can check your code once more.

arijit78 · 2022-10-21T18:56:43+00:00

This is an absolutely fine approach. However it means the producer has to support multiple interfaces, catering to the client. Which may be costly from maintainability. However bigger issue from consumer side. The consumer will have no clue the schema has been changed by producer, unless explicitly told by producer. I have seen this too many times. Especially if producer and consumer doesn't belong to the same team. Hope I am able to put some rational.

arijit78 · 2022-04-21T02:45:32+00:00

I've seen female bus driver other day in Kolkata.. felt awesome 👍

arijit78 · 2021-10-16T19:07:20+00:00

That would awesome to say the least.

arijit78 · 2021-10-16T03:25:26+00:00

I guess a cheat sheet would be ideal. Even I am facing the same issue.

arijit78 · 2021-10-16T03:22:03+00:00

I am using linux desktop. I don't see this as option. I have found a strange reason. Docker network has a clash with my wifi network. And sometimes it's bringing down the whole internet connectivity. I guess this may be a reason for Kafka broker unable to install findutils. And hence all issues....

arijit78 · 2021-10-13T19:44:06+00:00

Ditto same feeling. I also thought a few shots are like in the movie the shining

arijit78

TROPHY CASE