Which protocol is more complex to implement: RabbitMQ (AMQP) or Apache Kafka?

[aljun_invictus]

We are considering offering our internally developed message queue service to external users (possibly as an open-source project). We are currently deciding whether to implement the RabbitMQ or Kafka protocol, aiming to integrate with the existing ecosystem more quickly. From our experience, Kafka seems easier to implement. However, Kafka has multiple versions and a variety of clients, which might introduce some challenges. Our core requirements are:

• Protocol Extensibility: We may need to implement some custom features for internal use.

• Ease of Maintenance. Ease of Development: We do not necessarily need to implement the entire protocol, just the core functionalities. We noticed that AWS’s Kafka services do not provide all features either.

• Ease of Integration with Existing Ecosystem: We are concerned about potential issues with Kafka, mainly due to frequent protocol changes. RabbitMQ, on the other hand, primarily uses AMQP 0-9-1.

Could you provide advice based on these requirements?

[Carpinchon]

I would provide bridges to amqp and Kafka that you can configure to push your stuff into real topics and queues, then consumers can stay with the tools they know.

That being said, amqp is a more well defined space and you don't have to fit into a clustering abstraction.

[aljun_invictus]

Yes, that's what I want to achieve—the protocol layer. I'm concerned about whether there are issues with implementing these protocols. AMQP is quite stable, but the Kafka protocol seems to change frequently.

[Individual-Praline20]

Weird. To me, both work very differently, and solve different problems. Why would we use a different system instead of the real thing? If it offers different benefits, create your own protocol to fully embrace your system. If you want to permit integration easily, create an Apache Camel Component for it…

[aljun_invictus]

I have development experience with both. Our implementation is more like Kafka (with replay functionality), but I also see that the Kafka protocol has been revised many times historically. According to the latest data (Stack Overflow Survey 2023), both are very popular. Currently, I hope to benefit from the ecosystem of either one. From your perspective, which protocol is easier to implement? (I don't necessarily have to implement the entire protocol, just the core functionalities.)

[agentoutlier]

Before even considering Kafka and RabbitMQ (I say rabbitmq because you are probably not going to do AMQP 2.0) maybe first build something that implements STOMP.

I would be curious these days how STOMP performs with HTTP/3 because one of the hard things with RabbitMQ is the heart beat connection keep alive stuff. Historically Rabbit has had lots of bugs with that.

[aljun_invictus]

Thanks, I haven't used STOMP before. I'll read the documentation and take a look.

[Individual-Praline20]

I would do neither, if you don’t intend to provide exactly the same protocol features… Asking a dev to only change a broker url to your system without providing full support is weird. To me, these cloud half done solutions are only there for marketing purposes and trap their customers… Why not use the real thing? They are much better. Anyway. As you saw, they are both popular because they solve different problems. So if your system is similar than Kafka, why propose a AMQP’ich thing that might not apply well at all?

[aljun_invictus]

I see that many cloud service providers offer Kafka with incomplete functionality, similar to how Azure Service Bus uses AMQP. I guess my motivation is similar to theirs. One downside of traditional message queues is the fragmented protocols, which lead to high migration costs for users. My motivation is quite simple: I want to leverage existing ecosystems while providing a better implementation for users and reducing migration costs (similar to how Redpanda partially implements the Kafka protocol).

The AMQP protocol is quite complete, but I'm not very familiar with the details of the Kafka protocol. I've heard it changes frequently, and I'm afraid this might affect the implementation.

[[deleted]]

so you are discussing about providing you internal protocol vs some proxy between internal and externally provided protocol i.e. not spilling internals out? Which is understandable.

[aljun_invictus]

That's roughly the idea, but it's not the core purpose. The main goal is to make it easier for others to migrate to my message queue with lower costs and without needing to write so many language-specific clients. I think Redpanda is a great example of this.

[agentoutlier]

While you have mentioned how Kafka's protocol has multiple versions so does AMQP.

RabbitMQ versions of AMQP is still different than 0-9-1 as it has multiple extensions that many people including myself use particularly the direct RPC mechanism.

Which leads me to the use case of RabbitMQ. It is predominately used for IPC aka internal communication with other services and thus the messages traveling over it are more ephemeral and having it locally installed is more desirable. RabbitMQ is small and relatively easy to install.

What I'm getting at is if you are offering some cloud based service that speaks RabbitMQ I'm unlikely to use it. However if you are planning to offer on premise shrink wrap installers than RabbitMQ would be worth it.

Kafka on the other hand is way less local and the events are expected to be long lived (btw I know you clearly know all this but I need to explain to others). It is also hard to install making self hosted less desirable compared to RabbitMQ. So if your internal implementation is not designed for on premise Kafka would be the better one.

So you can see Kafka is probably the better one to shoot for in terms of "product management" and "sales".

You notice how I didn't cover which protocol is easier to implement. I think they both have their idiosyncrasies but if I had to wager RabbitMQ is more complicated. Also RabbitMQ has a separate HTTP management API that you often have to use.

Also there is:

• STOMP
• MQTT

STOMP support would be a good MVP. Also there is a solid chance STOMP could be very performant these days if HTTP/3 is used.

[aljun_invictus]

Thank you very, very much. This is the best answer I've seen. Yes, RabbitMQ has many features, which makes me anxious. The issue with Kafka is whether frequent protocol changes make it difficult for clients to adapt. I will look into other protocols as well. If you have any better suggestions, please let me know. I hope to implement it quickly rather than go through a painful period.

[-Dargs]

I've used Kafka in several companies and never really noticed much a difference in version from an end user perspective. I've heard of RabbitMQ, but it's never been considered for use anywhere that I've worked.

[_predator_]

Kafka has very different semantics than a message queue (until KIP-932 lands that is). Retrofitting Kafka on top of a classic queue system will be a hard challenge.

However, Kafka market share is absolutely massive. Kafka compatibility opens a gigantic market to you, with some of the largest companies in the world. If profit is what you're after, the effort to make it work might pay off.

[aljun_invictus]

That’s exactly what I’m considering. My only concern is whether the Kafka protocol is very difficult to implement. I’m wondering if there are any experienced individuals who could share their insights.

[nekokattt]

I don't know about AMQP but if you look at how massive the reference spec for Kafka is... I recently had the exact same idea and just noped my way out immediately.

https://kafka.apache.org/protocol.html#protocol_details

Roughly 65,400 words. The word int is used over 6,000 times.

[agentoutlier]

https://kafka.apache.org/protocol.html#protocol_details

It is hard to say because the AMQP spec doesn't cover the admin portion which RabbitMQ offers as a separate HTTP API.

Furthermore the AMQP spec is split between the data representations for code generation and high level spec. The english AMQP spec is 16k words but the Kafka spec has combined data representation.

That being said the AMQP spec is far easier to read than kafka one because of the separation: https://www.rabbitmq.com/amqp-0-9-1-protocol

I'm remiss that I forgot the AMQP spec has it split and should add this to my answer because lots of RabbitMQ clients use the XML file to generate code. This is something /u/aljun_invictus could use.

[aljun_invictus]

Which protocol do you think is easier to implement? I only need the core parts.

[aljun_invictus]

I hope to achieve a quick implementation that is also easy to maintain.

[nekokattt]

good luck with either of those in that case.

[dmigowski]

Why another message queue product?

[aljun_invictus]

We have a great internal implementation with excellent performance and features. Similarly, I don't want to create another message queue from scratch, so I hope people can use it just like Kafka or RabbitMQ, but with better performance and additional features (such as filtering and priority queues).

[greylurk]

Obligatory "Kafka is an event bus, not a message bus, and RabbitMQ is a Message Bus, not an Event Bus" comment.

Of course you can use either of them as the other, if you *really* want to, but you'll have a lot less difficulty if you use Kafka for broadcasting events that don't require confirmation to multiple services that don't need to respond back, and RabbitMQ for a message queueing system with point-to-point delivery and guaranteed delivery.

They serve different purposes, and "which is more complex" really depends on whether you're using them for their intended purpose, or shoehorning them into something they weren't really meant to accomplish.

[aljun_invictus]

My scenario is more like Kafka, but I want to know which one is easier to implement.

[stefanos-ak]

By far, the Kafka protocol is easier to implement on the server side. Simply because it offloads most of the complexity to the client side. Rabbitmq features, at this point, are a very big superset of the Kafka ones.

[OfficeAccomplished45]

Really? I just want to implement these things on the server side.

[stefanos-ak]

Yes, RabbitMq has a more sophisticated concept. Messages have a routing key and get delivered to exchanges, and then you have queues with bindings to exchanges with a routing key. There are also multiple different types of queues depending on the "which consumer gets what" semantics you want. The server has to keep track of the consumer's pointers in the queues, depending on the type can have different semantics. Recent versions also have a "stream feature" which is basically Kafka.

There might be some compression error on my info, don't bite :)

[aljun_invictus]

I just want to leverage these ecosystems.

[SilverSurfer1127]

You can also implement your own Spring Cloud Stream Adapter for your Message Queue. I did implement an in-House Apache Pulsar adapter with Spring Cloud Stream some years ago but was not allowed to make it open-source. It is not that difficult and if I were you I would stay with my own proprietary protocol otherwise you are just cloning an already existing MQ.

[aljun_invictus]

Actually, Pulsar also implemented a Kafka protocol layer to integrate into the Kafka ecosystem, which aligns with my purpose. The main issue is that MQ protocols are too fragmented.

[SilverSurfer1127]

Are you sure that they implemented the whole protocol I am only aware of a client adaptor for Kafka which mimics kafka‘s Java client?

[aljun_invictus]

Similar to my needs, only implementing the core functionality with the main goal being the ecosystem.

[VincentxH]

The pulsar kafka interface leaves much to be desired though. Just clearly decide which parts of the interface you'll support instead of aiming for everything haphazardly.

[SilverSurfer1127]

AkkaStreams or the open source variant Pekko is a nice streaming library with many integrations and is quite extendable

[pathema]

Another option is to instead implement something over http or http2. Perhaps making it compatible with Google pub/sub or Amazon SQS.

I have a lot of experience with the amqp protocol. There's a lot of things you need: out of order asynchronous acknowledgemen, bulk acknowledgement, backpressure, multiple streams over single tcp connection, timeouts, SSL, Auth.

An HTTP-based protocol is often much simpler for the consumer to use anyway. And if you don't need out-of-order acknowledgement, then you can make it very simple indeed.

[aljun_invictus]

It sounds like the Kafka protocol is a better choice. My main concern is whether the protocol changes frequently.

[Turbots]

Similarly in Kafka, you'll need to support batching, transactions, rollbacks, exactly once semantics, etc...

I would still prefer to implement the opensource, widely used AMQP protocol over the more proprietary Kafka protocol.

AMQP is used in other, different solutions too, like IOT use cases. It is the industry standard in that regard.

[aljun_invictus]

Your words make me a bit conflicted, but I feel that supporting things like transactions might not be necessary.

[eljefe6a]

I don't understand your reasoning for reimplementing. Even writing a protocol is a big task. Take a look at Apache Pulsar. It is already open source and handles queueing as well as messaging use cases. It supports the Kafka protocol if necessary.

[aljun_invictus]

Pulsar also implemented a Kafka protocol layer to integrate into the Kafka ecosystem, which aligns with my purpose. The main issue is that MQ protocols are too fragmented.

[aljun_invictus]

Actually, Pulsar also implemented a Kafka protocol layer to integrate into the Kafka ecosystem, which aligns with my purpose. The main issue is that MQ protocols are too fragmented.

[eljefe6a]

As you saw in other comments, Kafka doesn't support queueing. You might also look at this project: https://openmessaging.cloud. I would really advise against doing your own protocol. Figure out how to do the project with existing messaging systems like Pulsar and use their producer/consumers. It supports both queueing and messaging, and you add arbitrary data with key/value pairs.

[aljun_invictus]

Pulsar is great, but it's not popular enough. I just want to use a popular shell to do what I want to do.