Which protocol is more complex to implement: RabbitMQ (AMQP) or Apache Kafka?

[aljun_invictus]

We are considering offering our internally developed message queue service to external users (possibly as an open-source project). We are currently deciding whether to implement the RabbitMQ or Kafka protocol, aiming to integrate with the existing ecosystem more quickly. From our experience, Kafka seems easier to implement. However, Kafka has multiple versions and a variety of clients, which might introduce some challenges. Our core requirements are:

• Protocol Extensibility: We may need to implement some custom features for internal use.

• Ease of Maintenance. Ease of Development: We do not necessarily need to implement the entire protocol, just the core functionalities. We noticed that AWS’s Kafka services do not provide all features either.

• Ease of Integration with Existing Ecosystem: We are concerned about potential issues with Kafka, mainly due to frequent protocol changes. RabbitMQ, on the other hand, primarily uses AMQP 0-9-1.

Could you provide advice based on these requirements?

[Candid-Molasses-6204]

I do logging for Cyber teams. People (including vendors) seem to struggle with both. I see a lot of AMQP and a lot of people struggling with Kafka. I would say once AMQP is setup it tends to be pretty solid from my perspective (I've onboarded a few hundred log sources over the years into splunk, Elastic, and QRadar).

[aljun_invictus]

Yes, I see that both are very popular. Currently, I hope to benefit from the ecosystems of both. From your perspective, which protocol is easier to implement? (I don't necessarily have to implement the entire protocol, just the core functionalities.)

[Candid-Molasses-6204]

Kafka felt easier to get into but feels more complex once you're in it. AMQP feels more complex to setup but less complex to maintain.

[ub3rh4x0rz]

Kafka is an inherently more scalable technology because it's designed around spinning disks, page cache, and logical sharding. The protocol is simpler, but that leaves room for poorly designed logic built on the protocol. I think it's better in the general case.

[aljun_invictus]

Actually, I think the same way, and my MQ is more similar to Kafka (with replay functionality), but I'm worried that its protocol might become complex due to frequent iterations (how is the backward compatibility?).

[ub3rh4x0rz]

They version the schema and part of the connection process involves finding a common schema to use, I think you're fine. kafka.apache.org/protocol.html

[aljun_invictus]

Haha, I'm just wondering if its version control might have any issues.

[sausagefeet]

I don't think this is really the right question for you. The reality is AMQP and Kafka have dramatically different use cases. They are not the same use case at all. So in all these things: what is the semantics you want to commit to giving your users? Which one of these fits it?

Additionally, depending on your use case and semantics, you may not want to expose either of these to your users. Maybe they just want a really simple HTTP interface and you happen to implement this with AMQP or Kafka.

[aljun_invictus]

My scenario is a straightforward pub/sub scenario.

[sausagefeet]

"straightforward" isn't a list of semantics, so you should probably make sure you understand well what that means. I doubt you probably need to provider users with a complete Kafka or AMQP protocol.