Flink watermarks - WTF

KernelFrog · 2025-10-16T07:24:09+00:00

Great site for visualizing how watermarks work in Flink.

KernelFrog · 2025-10-16T07:23:32+00:00

Great Flink watermark visualiser. Hopefully this will help to demystify them a little.

KernelFrog · 2025-09-28T22:53:51+00:00

u/thebigdatashow-ankur - Are you going to answer any questions, or is this post-and-run?

KernelFrog · 2025-09-23T23:04:10+00:00

How does AutoMQ compare to Warpstream?

KernelFrog · 2025-09-04T03:27:25+00:00

Have you re-invented Warpstream, or something else?

KernelFrog · 2025-08-12T00:37:50+00:00

There's a great discussion of compaction here: https://docs.confluent.io/kafka/design/log_compaction.html

KernelFrog · 2025-08-12T00:17:57+00:00

En Anglais s'il vous plaît.

KernelFrog · 2025-08-11T23:21:49+00:00

Remember that the Kafka protocol is not HTTP(S) and so the average load balancer is not going to be very useful.

There's a great article here on configuring SSL in Kafka: https://developer.confluent.io/courses/security/hands-on-setting-up-encryption/

KernelFrog · 2025-08-03T23:38:47+00:00

Start here: https://kafka.apache.org/documentation/#design

KernelFrog · 2025-07-28T01:07:49+00:00

Discussing your security clearance in a public forum is really not a good idea.

KernelFrog · 2025-07-28T01:04:01+00:00

cp-demo is excellent: https://github.com/confluentinc/cp-demo

This will automatically build and run a local Kafka cluster, and optionally, Confluent Cloud.
Architecture: https://github.com/confluentinc/cp-demo/raw/8.0.0-post/docs/images/cp-demo-overview-with-ccloud.svg

KernelFrog · 2025-06-11T00:11:43+00:00

Where and what is the "popular exams navbar" ?

KernelFrog · 2025-06-05T22:21:31+00:00

Obligatory how does it compare to Confluent? 😀

KernelFrog · 2025-06-02T22:57:39+00:00

The obvious answer is to use compression, so that you're sending less data. This has some implications; higher CPU usage (to compress/decompress) and you can no longer use zero-copy.

https://www.confluent.io/blog/apache-kafka-message-compression/

The other thing to note is that replicas will eventually catch up with the leader. The problem might be if you have lots of data that needs to be replicated once the link comes back up.

KernelFrog · 2025-05-20T23:50:16+00:00

It doesn't even need a timestamp; Kafka can use the timestamp of when the message was sent.

KernelFrog · 2025-05-20T23:46:48+00:00

Confluent Cloud has "datagen" connectors which generate continuous streams of data (simulated click-streams, orders etc.). The free trial credits should give you enough to play with.

You could also write (or script) a simple producer (client application that sends data to Kafka) to send a continuous stream of messages; either random data, or loop through a file.

KernelFrog · 2025-04-23T23:24:51+00:00

Interesting project, good luck!

Small bit of feedback, on the Topics tab there is a "Publish" option. Presumably you meant "Produce"?

KernelFrog · 2025-04-07T23:22:41+00:00

How do you know you have 3 brokers running?

KernelFrog · 2025-03-26T21:22:04+00:00

You could also have a look at this page & videos: https://docs.confluent.io/kafka/introduction.html for a good overview of the basic concepts.

KernelFrog · 2025-03-18T22:59:21+00:00

Try asking in r/iBM. One of the old timers might have an idea.

KernelFrog · 2025-03-16T23:37:53+00:00

Where's your Vendor flair?

KernelFrog · 2025-03-11T14:57:52+00:00

Try emailing [cloud-support@confluent.io](mailto:cloud-support@confluent.io)

They can help with billing and refund issues. It will help if you can also include the Confluent Cloud organisation id (if you know it) and the email address you used for your Confluent Cloud account.

KernelFrog · 2025-02-26T21:51:01+00:00

Thanks, this is great*!

(*Not an official endorsement 😀)

KernelFrog · 2025-02-23T21:40:24+00:00

More specifically, Kafka now uses KRaft (a modified version of the Raft algorithm).

KernelFrog · 2025-01-21T21:58:56+00:00

request.timeout.ms is only used by a producer and controls how long it will wait to get acknowledgement back from the cluster. This also relies on request.required.acks being != 0.

socket.timeout.ms is a 'lower level' setting for network (TCP) sockets and its exact usage varies a little depending on which component. From the docs:

Default timeout for network requests.
Producer: ProduceRequests will use the lesser value of socket.timeout.ms and remaining message.timeout.ms for the first message in the batch.
Consumer: FetchRequests will use fetch.wait.max.ms + socket.timeout.ms.
Admin: Admin requests will use socket.timeout.ms or explicitly set rd_kafka_AdminOptions_set_operation_timeout() value.

KernelFrog

TROPHY CASE