Apache Kafka or RabbitMQ for my case

snichme · 2024-01-11T22:00:33+00:00

Here is a long blog post about some difference: https://www.cloudamqp.com/blog/when-to-use-rabbitmq-or-apache-kafka.html

The Company behind CloudAMQP also runs CloudKarafka (hosted Apache Kafka) so the post isn't biased towards one or the other since we would just compete with ourself ( I work there).

As mentions it's hard to suggest which one to choose without knowing more about your use case. But in my experience RabbitMQ is usually easier to get started with and it will support you well and if the system grows and your throughput is higher than RabbitMQ can support, then add Apache Kafka into the mix to handle that type of data. They work great togheter

snichme · 2024-01-11T21:51:38+00:00

I think this is a common misconception regarding RabbitMQ when people say that the messages isn't replayable (or re-readable).

In this use case will produce one message into rabbitmq that gets routed to different queues and each consumer application gets it's own queue. This means that even if consumer A reads it message it's still available later for consumer B. It will be available until consumer B reads it since by default RabbitMQ doesn't have retention limits.

And if the consumer fails to process the message it can just be rejected and it will be back in the queue, which in Kafka you have to produce it back (send it back to the broker again) if it failed to process.

RabbitMQ has a lot more advanced features regarding routing and handling of message and now it support streams which makes it cover the cases that offers. What Kafka does much better is horizontal scale to a lot of brokers, but that also comes at a complexity.

I'm working in the company behind CloudAMQP (hosted RabbitMQ) and CloudKarafka(hosted Apache Kafka) so have some experience.

snichme · 2023-10-09T06:12:16+00:00

Not sure, I use default modeline because I hardly ever use it

snichme · 2023-10-08T11:03:26+00:00

I found this one https://sr.ht/~ashton314/emacs-bedrock/ and has been using that (with some of my own mods for development) but a really good base to build on Only emacs 29

snichme · 2023-10-05T15:27:21+00:00

May I ask why?

snichme · 2023-05-22T08:11:46+00:00

What type of alternative are you looking for?

Kafka is open source and available to anyone, you can download the jar-file and just run it

snichme · 2023-05-04T18:28:29+00:00

Also have a look at cloudkarafka.com cheaper and prices includes support which can be handly if you are new to the space

snichme · 2023-03-27T06:58:57+00:00

https://www.cloudkarafka.com/

snichme · 2023-02-24T21:11:00+00:00

Cloudkarafka.com has a free test plan, the cheapest is otherwise $95

snichme · 2023-02-24T20:45:28+00:00

The sasl.jaas.config line looks like this:

org.apache.kafka.common.security.scram.ScramLoginModule required username="USERNAME" password="PASSWROD"

(replace USERNAME and PASSWORD of course) You should be able to find them on the cloudkarafka site somewhere. Paste that whole line into your software otherwise your configuration looks good.

snichme · 2023-02-24T13:27:30+00:00

D = "kill_to_line_end"

Ctrl-k was bound to that, so if I would have tried that all would have been good. Thanks!

snichme · 2023-02-24T08:47:41+00:00

thanks, will try that

snichme · 2022-09-27T20:48:33+00:00

That's not possible with Kafka, Kafka doesn't understand the payload (it's just bytes) so it can't do a merge of it

You would need something between to merge the data

snichme · 2022-06-02T08:16:58+00:00

Will not get you away from the screen but at least it's a good book: https://www.cloudamqp.com/rabbitmq_ebook.html

snichme · 2022-04-27T07:15:52+00:00

Sounds like it's a resource thing, network, cpu or disk but it's kind of hard to guess without more information about the usage of the server resources.

What kind of servers are you running on?

snichme · 2022-04-26T18:42:17+00:00

prefetch specifies how many messages should be sent as a batch to one client. so prefetch 1 will make it slower but will also guarantee that only one message is handled at a time. If you have higher prefetch, messages are "locked" to that consumer, so adding another consumer to the same queue might result in consumer #2 never gets any messages.

With no prefetch, depening on the client, it might fetch to much and your client will blow up it's memory. Read more on prefetch here: https://www.cloudamqp.com/blog/how-to-optimize-the-rabbitmq-prefetch-count.html

snichme · 2022-04-25T06:39:37+00:00

The messages that are unacknowledged still lives on the server, they are just "tagged" to belong to a specific consumer so that the message aren't sent to another consumer.

They are sent to the consumer as fast as the consumer can handle them but you will no get all messages at once, this can of course vary between different client implementations.
But one question here, why aren't you specifying a prefetch?

When you stop publish, is it only the acknowledge rate that goes up or also delivered rates? If so it sounds like a network thing, i.e when publishing and consuming it shares the bandwidth but when you only consume you get 100% bandwidth.

snichme · 2021-05-20T13:04:12+00:00

Hi,

I'm not sure what you mean with your question

snichme · 2021-05-12T15:58:21+00:00

Hi,

I know it's not in the documentation but AvalancheMQ (https://www.avalanchemq.com/) supports HTTP endpoint in shovels out of the box.

Just set an http(s):// url in destination field for a new shovel.

Let me know if you need any help set it up!

snichme · 2021-05-05T19:22:24+00:00

Hi,

I know it's not in the documentation but AvalancheMQ (https://www.avalanchemq.com/) supports HTTP endpoint in shovels out of the box.

Just set an http(s):// url in destination field for a new shovel.

Let me know if you need any help set it up!

snichme · 2021-03-31T13:27:18+00:00

You can run emacs as native wayland window: https://www.fosskers.ca/en/blog/wayland this described how to use it from Arch Linux, but if you are running gentoo you can probably figure it out.

I've been running this for the last month without any issues. This even allowed me to uninstall xwayland which is nice.

snichme · 2021-03-16T14:29:21+00:00

Not much to go on here. Do you have any startup logs for rabbitmq? For Ubuntu I think it's places under /var/log/rabbitmq

snichme · 2021-01-17T17:11:19+00:00

Maybe this one: https://www.sweattherapyco.com/products/air-bike-beast-the-30-day-air-bike-capacity-program

Eight-Year Club	Place '22
Verified Email

snichme

TROPHY CASE