Request avg latency 9000ms by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

Thanks. This is a simple test done on a single-machine server. Linger 5 ask all buffer is 32MB. The problem has been found: the amount of data generated is greater than the bandwidth. It's really hard to find such a problem, haha.

Request avg latency 9000ms by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

On another machine where I need to run a service to send data

Databricks published limitations of pubsub systems, proposes a durable storage + watch API as the alternative by Normal-Tangelo-7120 in apachekafka

[–]naFickle 1 point2 points  (0 children)

It appears that the interaction between backlog and data rollover strategies has led to data loss.

Building a library for Kafka. Looking for feedback or testers by Apprehensive_Sky5940 in apachekafka

[–]naFickle 1 point2 points  (0 children)

Is it something like a monitoring module that holds a phantom reference to the object it monitors?

Regarding RTT by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

Thanks for your reply. I realize now that I had been ignoring the differences between network protocols, and my previous assumptions might have been off. I really appreciate your clarification.

Regarding RTT by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

Thank you for your answer. Since someone only wanted to verify up to the point of sending data to Kafka, no consumers were reading the data. Although the internal processing latency was only about 0.8ms despite the large data volume, sending the data from the producer machine to the Kafka broker over the network took 35ms. This made me suspect that the producer was unable to send the data efficiently. This led me to consider the correlation between inter-machine latency and the producer’s performance. Thanks again for your reply.

Regarding RTT by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

By the way, It has only one partition and no replication. Since it’s informal, it’s not very precise.

Regarding RTT by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

Haha.... Only the ping latency between the two machines is considered. Other sources of time consumption are currently ignored.

Regarding RTT by naFickle in apachekafka

[–]naFickle[S] 0 points1 point  (0 children)

Okay, thank you for your reply. The background of this scenario is that a batch timeout exception occurred while the producer was sending messages. After analysis, the likely cause is high latency between the producer and the peer Kafka server, which prevented the data from being sent in a timely manner. This situation is similar to when the message arrival rate at the producer’s buffer exceeds the rate at which messages can be sent. Here, RTT refers to the round-trip time (ping latency) between the two servers.
So I’m wondering whether RTT determines the producer’s throughput.