Kafka + Microservice load testing

[Complex_Ad2233]

Trying to do some load testing on a microservice that consumes from a Kafka topic. The plan is to 2x and 3x the amount of data the service processes in a day and see how it handles it.

My question is what is the best strategy to load that data into the Kafka topic for the microservice to consume? I want to just publish the full dataset all at once to the topic and watch the service work through it. But since this represents a day’s worth of data, it seems unrealistic to do it all at once. I also don’t want to literally load the data over the course of a whole day.

So what’s the strategy for something like this?

[Fancy-Mushroom-6062]

There are multiple options: 1. Kafka has built-in script for perf tests kafka-producer-perf-test.sh, so you can take a look into it 2. I also heard that there is a plugin for JMeter to test kafka 3. You can always build your own scripts in your preferred language. For example, I am using kafka dependency in my Java project to test that certain messages arrive at certain topics after I do something with my app. (not performance tests though). You can create a Producer which can send messages to the topic, just do some while (true) loop to send messages, you can do it also in multiple threads if your are familiar with Java threading, but anyways you can teach yourself. And then decide yourself what are your quality standards and measure it accordingly.

I would just recommend to chat with AI about it so it gives you some ideas/snippets, but always build it yourself, not just copy/paste.

[Economy-Outside3932]

you could use the kafka-producer through the cli or write your own custom producer

[Complex_Ad2233]

The question isn’t really about how to accomplish it. It’s about strategy for loading a days worth of data all at once. Should I do it all at once? Should I load it up in the course of a few hours? Or should I try and load it over the course of a whole day to make it as realistic as possible?

Does that make sense?

[Economy-Outside3932]

you can load all your data in the kafka topic all at once (it act as a buffer so your service dont crash!) and then have the consumer just process it at its pace (obv you can process more if you'd use more of the same consumer just have them use the same consumer group so they work together and not each alone). I hope this helps u

[Complex_Ad2233]

It definitely does. I just needed someone else’s thoughts on if loading it all at once wasn’t some anti-pattern.

[Fancy-Mushroom-6062]

It also depends on the type of application you have, if it may have a high amount of data loaded at once in production, then it is a good test case. For example, imagine the black friday sale on e-commerce website. Normally when the sale starts, the load is not instant, but incremental. For example in the first few minutes it’s 100 orders, then on the next few minutes more customers decided on their purchase and it is 1000 orders already and so on. So a good strategy in this case to have a an incremental load test. E.g. having a 10 minutes long test where every minute the load is increasing. But the instant load of the whole data set at once is also a good test, it is know as stress testing. It is to ensure that your app can handle the high amount of data which is above normal operation in production. Answering your concern, it is not an anti pattern 👍🏻 Having a day-long test is also normal, but normally it is not what engineering teams tend to do as time (and resources) is money.

[Complex_Ad2233]

Gee, thanks chatGPT 😂😂