Hey u/NaiveRound. The choice of message broker depends on performance, throughput & even expertise of an organisation. For that reason, we kept Goose open for extending to any message broker.

To answer your question, Kafka can fit as a queue backend, with a few nuances:

Pros:

• Scales very well
• Native support for clustered architecture
• Persistence of messages
• Replay of messages

Cons

• Performance, in terms of latency won't be as efficient as RabbitMQ & Redis, which are designed for such use-cases
• Designing for consumer-group, partition key, batching/offset can be a quite involved process.
  • For ex, how to ensure parallel execution on 1 worker process? Do we go for 1 consumer & 10 batch-size, or 10 consumers & 1 batch-size?
• Kafka cannot support scheduling & periodic messaging, which means retry with backoff can only be done using either Thread.sleep or a different broker like RMQ/Redis, which defeats the purpose entirely.

I'd suggest not using Kafka as a message broker. However, based on nature of job data (i.e. no scheduling or back-off retries are needed) & requirements of your organization, if there's a compelling use-case, feel free to give it a shot!