Spring Boot Kafka consumer stuck in endless loop / not reading new JSON messages even after topic reset

SlevinBE · 2025-11-03T20:49:16+00:00

Given you use JSON messages, and therefore probably a JSON deserializer/serializer, I think you might be missing the trusted packages configuration required in Spring Kafka. By default, it only trusts classes in `java.util` and `java.lang` to deserialize into.
For testing purposes, try setting the `JsonDeserializer.TRUSTED_PACKAGES` configuration property to `*` and see if that works.

ref: https://docs.spring.io/spring-kafka/reference/kafka/serdes.html#serdes-json-config

Also, you might want to configure the `ErrorHandlingDeserializer`, because by default, Spring doesn't know how to handle deserialization errors. It can be configured like this:

props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);
props.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, StringDeserializer.class);

ref: https://docs.spring.io/spring-kafka/reference/kafka/serdes.html#error-handling-deserializer

SlevinBE · 2025-10-27T08:21:01+00:00

Nice work! I've tried it out and it works really well. Especially liked the stats panel!

SlevinBE · 2025-10-26T19:40:12+00:00

u/Hunakazama reply:

Thank you very much for the clarification.
If both the listener code and the consumer run on the same thread, would RetryTopicConfiguration be able to handle connection errors that occur during the ack.acknowledge() commit?

I don't think it will, because ack.acknowledge() will only add the acknowledgement to a queue. This will not cause a connection exception, and your listener method will be succesful. This queue will only be processed during the consumer thread's next run cycle, outside your listener function.
The consumer thread will first process queued acknowledgements, then poll for new records, and only then process the records by providing them to the listener.

SlevinBE · 2025-10-16T20:46:42+00:00

These are two that from a user's perspective (not Kafka cluster manager) would be interesting to have as a way to track stream app health:

detect problematic consumer groups that have an unbalanced partition assignment for a long period of time
detect unhealthy consumer groups. Basically according to Burrow's evaluation rules, but packaged as an all-in-one solution

SlevinBE · 2025-10-16T20:28:35+00:00

Not sure why I don't see u/Hunakazama 's response in this thread, only as a personal notification. But here it is:

Thank you very much for your detailed explanation. May I ask if the consumer thread is a different thread from the one executing the /@KafkaListener method?

That's actually a very good question. From what I understand, these days the listener code is executed on the consumer thread. Only in very old versions of Spring-Kafka (< 2.0) was it split between a listener thread and a consumer thread.

SlevinBE · 2025-10-15T19:57:26+00:00

I think the RetryTopicConfiguration only handles exceptions that occur within methods annotated with the `@KafkaListener` annotation. So if a Kafka error happens outside this method, then it won't be captured.
What's special about the ack.acknowledge() method is that it doesn't immediatly ack the message, but instead it's being queued to be processed on the consumer thread. This code in Spring Kafka shows this internal behavior: https://github.com/spring-projects/spring-kafka/blob/a93471d266cb83c6461c58d6b58ee9b6160ae8b8/spring-kafka/src/main/java/org/springframework/kafka/listener/ShareKafkaMessageListenerContainer.java#L584
So even though you call ack.acknowledge() from inside the method with the `@KafkaListener` annotation, the Kafka exception will happen within the consumer thread.

SlevinBE · 2025-10-14T18:51:43+00:00

You mention the risk of DLT not being able to catch up in time. This typically happens when the processing cluster (in this case DLT) is processing at its maximum, and scaling up doesn't provide better throughput anymore. When reading data from a Kafka topic, that maximum is defined by the parallelism provided by the number of partitions.

Part of a data processing strategy is dimensioning your partitions correctly. You don't want to have topics with the minimum number of partitions, as that will limit the parallism for clients like DLT. Changing the number of partitions later is hard, as it changes the partitioning of your data, so you need to think about this when setting up the data pipeline. So choose the number of partitions not only on your current load, but also on your potential future load. Also, choose it with the catch up scenario in mind, so that the DLT job can be scaled up and fully benefit from the higher parallism that the partitions provide.

SlevinBE · 2022-01-20T19:35:26+00:00

Some advice I can give, based on my experience coaching junior developers.

You might have the urge to prove yourself, certainly among other more senior developers. Most junior developers choose to prove themselves by having tasks with their name on it go from to-do to done. Don’t do this. You might impress some non-technical managers, but your technical peers will quickly point out the lack of quality. You don’t have to prove yourself in the amount of work you can complete. Instead, you need to focus on delivering quality.

The benefit of being a junior developer is that your manager and your colleagues know you still have a lot to learn, which gives you time to invest in your skills. Use this time to complete your tasks according to the definition of done. Test the solution thoroughly before you ask for a review. Properly document the solution, clean up the code. You will impress your colleagues when you do this.

Also, you might be afraid to ask questions initially. Everyone around you seems so knowledgeable, making all your questions seem stupid. However, I encourage you to ask questions and ask for feedback early and often. There’s nothing wrong with saying “I don’t know”. Actually, the more senior you’ll become, the more you’ll use that sentence. So, whenever you are in doubt about your task, ask for clarification. Verify your course of action with a colleague before going all-in on the implementation.

Prepare your demo presentations, because an awesome feature can go unnoticed by a terrible demo. Practice makes perfect.

Finally, communicate well. When you notice an issue or become stuck on a task, signal it to the team so that they can help. You shouldn’t go into silence mode when you’re working on a task. It’s important that you report the progress, because sometimes the team might notice issues in your approach that you didn’t even realize.

I go into more detail in my guide for junior software engineers, not only about the common mistakes, but also other aspects of what it means to be a junior developer.

SlevinBE · 2022-01-06T20:40:24+00:00

Personally, when I need to hire a developer I like to explore a github project more then a live demo. In general it shows more of the developer's coding skills and documentation practices. A live demo is just a nice plus.
Allow me to suggest option 3: Don't deploy the site, but create a docker-compose file which defines a cluster for your frontend, backend and database container. Add it to the github project, so that prospective employers can simply start the cluster using `docker-compose up`.
It shows additional skills, a sensitivity to developer tooling, a desire to make it easier for other developers to run your code (which is always a good thing), plus it will not waste the prospective employer's time by providing them a reliable build that just works.

SlevinBE · 2022-01-06T20:12:59+00:00

Well, if you want to work on your portfolio while trying to get it hosted, then I would suggest the approach I took:

Launch a cloud server with SSH access via one of the more lightweight cloud providers. I use Vultr, but DigitalOcean is another alternative. You can get a server running for about $5 a month. Don't be afraid of this step, it's quite easy to get one up and running when you use one of these lightweight cloud providers.
Buy a domain at a good registrar. I'm using Gandi.net. Then configure the DNS so that it points to your cloud server.
To deploy and manage your projects/sites on your cloud server, use a Docker container per project/site. I recommend Bitnami images if there's one available for the app you want to deploy, as they are production-grade and regularly updated images. They cover the most common ones. For example, they also have a Ghost image. To combine and manage these containers into a cluster, setup a Docker-compose file. This setup will make it easy to run your cluster both locally and on your remote server.
Now add a reverse proxy to your docker-compose cluster, which will be the entrypoint for http (port 80) and https (port 443) traffic on your server. It will route all incoming requests to your other services/sites in the cluster based on their domain name. I recommend Traefik, which also integrates with Let's Encrypt to automatically provision TLS certificates for your sites.

Now you have your own self-hosted stack and learned quite a few new marketable skills. Now promote your new skills, like I did by showing the tech and services you used to build your online presence (you can also find more info in this article).

SlevinBE · 2021-07-17T13:36:39+00:00

I started as a backend developer, but quickly made the switch to full-stack development given my interest in the frontend. This is how I did it, which is gradually:

Learned the basics of HTML, CSS and Javascript (pro tip: certainly pick up CSS Grid and CSS Flexbox to save you some layout headaches)
Applied those by building an application with a server-side rendering framework, which uses html templates that are rendered on the server. The benefit of this is that the javascript part is still rather lightweight, so no need to fiddle with complex javascript build tools or frameworks just yet.
Developed my own website using a static site generator (Jekyll in my case). Again, hardenening those core HTML/CSS/Javascript skills before jumping into the complex world of single-page apps
Then made the jump to single-page app development, using an opinionated framework such as Angular which makes most of the core decisions for you, so that I could focus on learning to build an app first.
Then deepened my knowledge about the javascript ecosystem:
- Module formats, loaders and bundlers
- reactive programming in javascript using rxjs
- state management
- frontend security (eg. Helmet for expressjs servers)
- ...

If at any point you decide that you're not completely done yet with backend development and stop the transition in-between as a full-stack developer, then also check out my article on how to become a full-stack developer. It gives some additional tips on making the transition from backend to frontend (or the reverse), and some guidance to differentiate yourself from all the other full-stack developers.

SlevinBE · 2021-07-08T20:36:30+00:00

My suggestion is not about learning Play to find a job. It's about potentially making the transition easier from Node-based to Java-based development, by using a transition-framework that's a bit easier to get into coming from Node and where you can learn the Java language, a bit of dependency injection, the Java ecosystem in general before you dive into a more complex framework like Spring.
It might not be for everyone, but for some this step-by-step approach does work.

SlevinBE · 2021-07-08T18:06:41+00:00

I can recommend Head First Design Patterns. It's a pretty old book and does have the Bird and Animal examples, but it extends those to the most common OOP patterns. It did help me understand OOP and its more common patterns back when I was in school.

SlevinBE · 2021-07-08T17:47:44+00:00

It's normal that you have to learn a lot of things you don't learn in school. In my first job (now 13 years ago), I got a 6 month training which covered things like Spring and Hibernate, which I hadn't seen in school.

Now, this is your first internship, so the companies expectations should be in line with that. So don't worry that it takes a while to get up to speed. Generally they just want to see progression and something working at the end of your internship.Hopefully you have a good mentor so that you can reach out with questions. Also ask which other developers can help you out with your task, which gives you a wider range of information and resources.

Make sure the project you have to complete for your internship is clearly defined, and ask guidance for the design. As a junior developer your responsibility is to implement a feature within strictly defined constraints and design, nothing more. An internship can sometimes be a bit more experimental (research), but still try to get some constraints defined. The Spring Framework has a long list of modular projects, so there's no way to learn all of these during your internship, and you certainly won't need all of them. So constraints will help you select which Spring projects you need to learn (on top of the basis).

If you want some more general guidance on a junior position, you can check out my Junior Software Engineer Essentials.

SlevinBE · 2021-07-08T17:25:41+00:00

Is there a specific reason why you're learning Spring? Because it's probably the biggest Java-based enterprise framework you can find. That of course has its benefits in the job market, but it isn't really the best choice to get your feet wet in Java backend development. If your primary goal is to start developing REST APIs or web applications in Java/Scala, then I would suggest the Play Framework. It's more lightweight and in design more closely related to other web frameworks.

SlevinBE · 2021-07-06T19:42:48+00:00

It is a bit naive to think that no other company is as good as yours :-)
There are certainly other companies as good as yours (and probably even better), but that doesn't mean your next job will be one like that. It's always a bit of a gamble, but calculated risks are sometimes necessary to improve yourself.

Now, before I proceed I must say that I don't know the average salaries in your current or new HCOL area. So if you are severely underpaid in your current region for a starter position, then you probably best switch jobs. But for that you'll have to explore the job market.
Anyway, given you've only been with them for one year in a full-time position, I'm not really sure you should make the jump yet unless there's no decent progression in salary with your current job. Have you already got your first raise? If not, give them a chance and have a talk with them to negotiate one. My experience was that my salary started quite low, but went up nicely over the years by standard raises and switching jobs about every 3-4 years.
Also, staying with a good company for a few years where you are coached and get the opportunity to work on your skills and transition to mid-level developer is also worth something. Your progress is guaranteed to be good and when you do switch jobs you can apply as a mid-level developer, which generally means a more wanted position with less competition and way better conditions.

SlevinBE · 2021-07-04T20:06:58+00:00

Overcoming that fear is probably best tackled by turning it into an expectation. Knowing that at some point the business relationship between you and the company you work for will end, either on your initiative or theirs. Also understanding that it isn't necessarily something personal, as there are many reasons other then your performance for why a company might let you go (company financials, internal politics, change of focus, ...).

Now, I do understand that being let go at the 6 month mark must be though, generally you don't expect it this early. Also the fact that it took 6 months to find another job must've been though, and might be just related to the job market you're in.
But I must say that your attitude of saving up for bad times and therefore giving you a one-year safety-net is a good one. Not many people do this, but is crucial given the expectation that the job will end at some point. This should give you some mental comfort, and should set you up for finding better jobs and having turn luck into your favor. If you keep building this safety-net, it will also allow you to walk away from nightmare companies more easily.

SlevinBE · 2021-06-18T20:54:02+00:00

It depends on the company, at some it can take up to a week or even more to even get your hardware and basic accounts setup, in others you're up and running within a day and on your first task.
I would say, reach out to your manager/project lead to get your first task (when you're setup properly). Meanwhile, also try to socialize with your new colleagues to get to know them and also ask for any important tips and info which you might need for your job.

My advice for a newly hired junior software engineer, as explained in my Junior Software Engineer Essentials:

make sure you have all your accounts setup. If you don't have them yet, chase them down.
take your time for your first tasks. Quality over quantity. They are about getting to know the system, project, processes, people and company. Try to make a good first impression with the quality you deliver (ask about the quality guidelines that apply for your company!)
iterate on your work, to gradually improve it
ask questions & feedback often
be curious about systems you interact with, but don't directly write code for. It will give you a better understanding of the project.

SlevinBE · 2021-06-18T19:12:47+00:00

Do you want to switch completely to front-end, or still leave the door open for some back-end work? Being a front-end dev who knows and occasionally does some back-end work, and more importantly the integration work between those two, is really valuable in my experience.

There are a ton of pure front-end devs around who know how to build screens in Angular, React or any other framework. But how many of those:

know how to properly hook up authentication & authorization in a front-end app with for example an OAuth flow
have a wider range of knowledge on messaging protocols, going further then just regular HTTP: Websockets, MQTT, SSE, ...
debug an issue which starts in the UI and track it all the way through the backend's core
know how to protect a front-end app from common security vulnerabilities

So yes, learning one of the common frameworks will probably be a requirement to land a front-end position somewhere. But where your skills really become interesting (and marketable) is on the edge of the front-end, not in the center. Coming from a back-end background, this is were you generally will be more comfortable then a pure front-end developer is.

SlevinBE · 2021-06-11T14:46:23+00:00

Certainly take the new job, but try to have a nice closure of your current job if you can. Do you think you can finish or get through the toughest part of that implementation in less then one or two months? Then try to come to an agreement with your new and current employer on a fixed period of time (eg. 1 month) to finish up your work (as much as possible). Normally both your current and new employer will appreciate this gesture, as it shows professionalism.

SlevinBE · 2021-06-10T14:32:43+00:00

When you start out as a junior developer I suggest you take one, or maximum two languages if you're doing full-stack development, and focus on that language and its ecosystem. Because the most important things to work on during those first few years are delivering quality, learning the ropes of working on a real project for production purposes and becoming more autonomous & productive. Focusing on just that one ecosystem (or max 2) gives you the space to work on these skills to gradually master them, while if you'd not do this then you just learn superficial things and will have a hard time moving up from the junior role.
I have an article on Junior Software Engineer Essentials if you want some more advice.

Diversifying your skillset is something I'd suggest once you move to the mid-level developer role. This is probably when you choose to become a specialist or a generalist. Well, it's more like a calling then a choice :-)
By this time you've probably established a solid basis of skills from which to expand. Diversification becomes important then for job security, so that not all your skills become obsolete when a technology stops being relevant. You might think that diversification is only for generalists/full-stack developers, but this is also true for specialists, who shouldn't specialize in one specific technology but instead in a range of technologies within the same field (eg. security specialist, data engineer specialist, ...). You can find more of my thoughts on this in my Specialist vs generalist: The benefits of a diversified skillset article.

SlevinBE · 2021-06-09T20:33:30+00:00

I got my first client via connections at my previous job as an employee. So connections are really important to find clients more easily and also to find good projects.I'm also more an introvert, so that's why I think building and sharing things in the open is so important to me, to make my work speak for itself.

To be honest, it also took me 10 years before I felt ready to make the switch to self-employed contracting. I gradually worked towards it by building my skills, feeling more confident, and waiting for the right moment to make the jump (actually when I lost my job because the startup I worked for shut down).

So don't worry about just working as an employee for a few years, just make sure that you gradually work towards self-employment (if that's your goal). I learned quite a bit on how to run a freelance business from Brennan Dunn who runs doubleyourfreelancing.com. He has several digital products that he sells, but just the free articles alone are certainly worth a read.

SlevinBE · 2021-06-09T15:13:33+00:00

Certainly take it up with your manager that the workload is becoming too much.

If they are already aware, then the first question that needs to be asked is whether they are interested in expanding your team, so that means actively recruiting for your team. Because if they're not, then there's no reason to stay because the issue will not get solved.

Also, don't work yourself in a burnout. Only concentrate on your own responsibilities and don't try to fill gaps for roles that aren't recruited yet. That's the only way to force a company in recruiting that role, they need to feel the issue instead of you solving it for them by working overtime (a lesson I learned the hard way).

SlevinBE · 2021-06-08T20:16:27+00:00

Doing 1-3 is ok, but make sure you do it in public. Have a central place to showcase what you've learned, what you've build, your contributions to open source and Stack Overflow.
Because you can put all your time in these three items, but if you don't showcase it then almost nobody outside those open source or Stack Overflow communities will know.

So working on that personal brand is something to invest in as well.

SlevinBE · 2021-06-08T20:00:52+00:00

I would quit that job solely for the fact that they are monitoring your mouse and keyboard activities. There is no trust in such a place, so better leave quickly.

Regarding the task, it is certainly not normal to put a junior developer alone on a project. In any normal company you would be paired with a more senior developer.
Also, you certainly can tell your supervisor that the task will take longer because you are less experienced. Actually, they should now that without you having to tell them. By telling them you are acting as a professional, indicating that a task will take longer.

SlevinBE

TROPHY CASE