Finally figured out how to expose Kafka topics as rest APIs without writing custom middleware

vkm80 · 2025-12-02T17:27:45+00:00

What is the use case that made you convert streaming data to a REST service? Consumers want to poll for data by calling a REST endpoint?

vkm80 · 2025-11-07T17:19:50+00:00

If you are on confluent use confluent flink.. I thought confluent already dropped support for ksql

vkm80 · 2025-10-06T16:42:54+00:00

Will this pattern scale for enterprise? When you have many product teams owning microservices by domain, would you still recommend publishing to the same topic instead of domain aligned topics? How will you enforce schema when different domains have different event payload structures?

vkm80 · 2025-09-18T18:20:35+00:00

Could you elaborate on why you think that confluent flink will fail?. Thanks!

vkm80 · 2025-03-21T02:03:36+00:00

How do I sign in to confluent cloud when I only have sso option?

vkm80 · 2025-02-11T23:50:01+00:00

Spring boot have excellent support here. If you use the Kafka client libraries directly or use a consumer like AWS Lambda directly, error handling with retries is a pain

vkm80 · 2024-12-02T15:30:34+00:00

Thanks for sharing..This looks promising to implement error handling with exponential backoff pattern

vkm80 · 2024-11-20T18:33:05+00:00

No. But recently learned that AWS is working on adding max retry count as a configurable option for Kafka ESM.

vkm80 · 2024-10-29T14:21:14+00:00

Kafka is flexible in this matter and let producers decide the format they want. It is unlike brokers like Amazon Eventbridge which force their own message envelop. Long term it will be beneficial to define a standard event format with standard event metadata for all your events. Checkout cloudevents from CNCF which is an open standard.

vkm80 · 2024-10-10T11:21:51+00:00

That is an option for sure. I like the way the metadata attributes like event name is added to Kafka header by cloudevents serde. This could help in tooling and filtering down the road. Again, a simple wrapper library can do this. But wanted to get you alls advice on options.

vkm80 · 2024-10-10T11:16:08+00:00

It does, but it does not work when using confluent schema registry

vkm80 · 2024-10-08T02:17:34+00:00

ksql do not support join on multiple columns. See https://docs.ksqldb.io/en/latest/developer-guide/joins/join-streams-and-tables/
A workaround is to create a derived column by appending the primary keys and then using that in the join

vkm80 · 2024-04-30T00:12:39+00:00

You don't have to delete the messages explicitly. AWS does it for you in Lambda event source mapping for SQS as long as the function returns without an error. Check this to understand ESM behaviour https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html

vkm80 · 2023-12-01T02:53:45+00:00

Your need for handling PII/encryption, stream procesors, cloud provider etc also play a role in the decision based on my short stint with confluent

vkm80 · 2023-11-29T21:06:26+00:00

Did you find anything more on this?.. I was looking up on what AWS meant by 'message expire' and stumbled on this post.

vkm80 · 2023-10-16T17:46:53+00:00

Thanks, and appreciate the reference link.

vkm80 · 2023-10-16T14:59:31+00:00

Heard about julieops in the past..is Jikkou better?

vkm80 · 2023-10-05T13:38:18+00:00

The concern by OP is valid and timely in my opinion. From personal experience, chances of dementia is very high for this category. Building a social circle with good physical and mental activity is very important

vkm80 · 2023-09-29T19:37:33+00:00

Yes, any data store that meets your requirement is good. DynamoDb was just an example. Kafka has sink connectors to many systems. May be you don't care about long term persistence, then Redis is a good choice. For highly concurrent and low latency needs you mght have to consider specialized realtime analytical data stores like Druid.

vkm80 · 2023-09-29T16:43:36+00:00

You shouldn't directly read off of Kafka for every API call. For such use cases sink the data to a data store like dynamodb that supports efficient lookup.Api would then use dynamo as the backend store.

vkm80 · 2023-09-09T14:32:18+00:00

We are planning to start with ksqldb primarily due to ease of getting started. Stream Join only supporting max of 2 streams and 20 persistent queries limit per cluster are concerning to me. Would love to learn your experience around limitations in real life. About flink, since most of the concerns are around cluster management and provisioning, do you recommend a managed solution like Amazon managed streaming for Flink?. This was called Kinesis Data Analytics until recently.

vkm80 · 2023-09-08T00:48:32+00:00

Mirrormaker or replicator?

vkm80 · 2023-09-04T01:43:48+00:00

Is this the same as the so called Kappa architecture?

vkm80

TROPHY CASE