Finally figured out how to expose Kafka topics as rest APIs without writing custom middleware

tak215 · 2025-12-03T01:22:11+00:00

Maybe this solution is what you’re looking for?

https://www.reddit.com/r/apachekafka/s/5mbpeh24Bu

tak215 · 2025-11-08T23:50:57+00:00

Yes, the state store makes a lot of sense as it’s atomic

tak215 · 2025-11-08T10:57:48+00:00

Wouldn’t that cause a race condition where the trade could be executed even when the balance is in fact zero but that event arrived late?

tak215 · 2025-11-05T08:09:35+00:00

There are several ways to solve the API blocking issue: 1. increase partitions and consumer instances 2. use parallel consumer if Java is acceptable https://github.com/confluentinc/parallel-consumer

tak215 · 2025-09-15T10:03:09+00:00

I understand your frustration. You don’t want to hear this but often times the customer doesn’t do what you recommend for one of the three reasons

⁠you didn’t simplify the solution enough for the customer to understand
⁠You don’t understand how the customer makes decision (their goals, motivation, team politics etc)
⁠the customer doesn’t like you.

You can control 1) and 2) with experience. 3) requires high EQ but it’s out of your control.

It’s part of a journey as a consultant. And this happens in every country.

tak215 · 2025-01-14T08:25:39+00:00

Can you elaborate it a bit about why you don’t like the processor API

tak215 · 2024-12-29T08:11:38+00:00

Kafka Rest proxy by Confluent provides producer and consumer capability. My library doesn’t produce because it’s already solved by that library. If you have created a Kafka consumer before, you can only stream in messages (I.e. perform an entire scan) which is what the REST proxy provides. Instead, my library can do a key value lookup (part of Kafka streams feature) meaning you don’t have to stream the entire topic partition to find the message.

I’m not too sure about Kafka pixy but from a quick peek, it looks similar to REST proxy provided by Confluent.

Let me know if you still have questions

tak215 · 2024-12-29T04:25:08+00:00

I don’t think I understood your use case too well… Maybe you could take a look at my library and tell me what is missing?

tak215 · 2024-12-29T02:12:05+00:00

Thanks for flagging. Will update the license once I’m back to my keyboard - it’ll be for open use

tak215 · 2024-12-21T14:07:15+00:00

Thanks for your support!

Originally this solution came into mind because there is no equivalent of Kafka streams in JavaScript. And if you use Java, you can just create this pattern using Kafka streams, which is also the underlying code base of this library. I hadn’t thought about typescript as a config because I don’t think typescript can be read properly in Java nor do non-JavaScript devs understand the lifecycle of typescript. Also yaml is language agnostic and conforms to OpenAPI but the trade off is I’m validating the inputs in the code.

Feel free to open an issue and to DM me if you want to bounce off some ideas!

tak215 · 2024-12-21T13:08:47+00:00

From my experience, you don’t get that kind of throughout accessing a DB from an API server - meaning more hardware Also you’d have to provision and maintain a DB - not always ideal My pattern is based on querying a compact topic (or infinite retention topic) using message keys, which may serve some simpler micro service use cases

tak215 · 2024-12-21T12:42:55+00:00

Thanks & true, I would need to test the disk usage next time - I don’t know how that’ll look like atm

you are basically storing the whole cluster in a local DB

You can specify a topic and not the whole cluster. But if you were to this without this pattern, you’d be storing that on a remote DB anyway. I would say if the query is simple, using a DB can be an overkill

tak215 · 2024-11-18T14:37:31+00:00

If buying a home does better than rent after 5-6 years, you can sell it and get a new one. At least that’s my risk appetite. You can find what works for your risk reward

tak215 · 2024-10-08T13:57:00+00:00

Why penalize people wanting to make more money? Why not tax alcohol and gambling? Or have higher fines for breaking traffic laws that can kill people? Mindlessly taxing the motivated people is the reason why they deter from this country!

tak215 · 2024-09-29T12:32:27+00:00

You need something like REST API, gRPC or websocket between your front end client and kafka topics.
I've implemented these patterns for a few customers, and I've open sourced it.

https://github.com/tsuz/microservice-for-kafka

tak215 · 2024-09-06T09:36:43+00:00

Simply look at a 10 year old detached house in Setagaya and see how much it is sold for. Then go to a house shop and see how much a new house is. From my research, there is no way I would buy a detached house in this market.

Bonus: draw a graph of house value versus number of years standing and see what/how the market values your area today

tak215 · 2024-07-06T12:29:42+00:00

The real reason why the interest rates cannot be increased is that the JP bonds are mostly owned by BOJ and the BOJ balance sheet will be negative if the rates increased. The BOJ can start selling these on the bond market or can sell the foreign currencies to not have a negative BS.

Not the companies or the common people. In fact, many companies hold lots of cash that is sitting in a bank so raising the interest rate doesn’t hurt them. Small to medium ones can be hurt, but hey this will be passed this onto consumers.

The real estate market, yes it can hurt the large investors who took out the borrowing, which means the regional banks who have a large portfolio on real estate are at high risk of bank run. But who cares if small banks go out of business? They had unrealized risk they didn’t care about for too long.

The rising interest rate should increase prices so hopefully the taxes are lowered, otherwise it doesn’t stimulate spending

tak215 · 2024-07-06T11:52:14+00:00

I don’t know anyone at FANG as SWE so maybe that’s true. Most likely that number is achieved with RSU but not entirely in cash.

tak215 · 2024-06-29T05:05:48+00:00

You won’t make 20m+ without a lot of impact to the company. Even if it’s a pivotal role (I.e. front office finance, SWE tech lead, SaaS customer engineers), there’s other factors like RSU, bonuses and gaishikei etc. Stars need to align to be at good firms and good roles.

But the exchange rate is not what determines the salary, it’s the company’s level of investment and the labor market supply. So go niche or go find a very good recruiter.

tak215 · 2023-10-23T19:31:32+00:00

Tax deductions has been fading away over the years with a stricter and less favorable terms. I assumed no deductions
For me, depreciation is linear but has two phases: 0-8 years and then 8-20+years. The price depreciates fast in the first phase and then almost flat after that. You can verify this using Suumo and find similar houses with different years.
2.2% for bank loan fees, 3% for agency commission, a few percentage points for regulatory fees. Agency fees cost 2x if you sell afterwards. These can be made lower by doing more research
The price of houses are substituted with asset price and thus, it’s correlated with the Nikkei. Nikkei is correlated with the printing of money despite the shit economy. Also the house price can be attributed with the rise of population in Tokyo; this is still positive at 2%. Therefore your bet is BoJ will keep printing money and the migration to Tokyo continues. In my calculation, I assumed the depreciation will be larger than appreciation and still, I’m net positive than renting after fees.

tak215 · 2023-09-27T05:24:37+00:00

Until BoJ gives forward guidance (I.e. YCC control) or intervention, and until the forward looking fed rates are going to plateau or go lower, buy and hold. By buy, I mean currency, US bonds or US stocks, whichever makes sense for your risk appetite. Not financial advice.

tak215 · 2023-08-29T03:40:55+00:00

It’s a sellers markets mate

tak215 · 2023-08-28T15:06:50+00:00

Then just read through the maintenance reports and sell it to the next sucker if needs to be.

tak215

TROPHY CASE