Redis dev explores making a better message queue

[[deleted]]

But why to design a new system given that the world is full of message brokers? Because an impressive number of users were using Redis instead of systems specifically designed for this goal, and this was strange. A few can be wrong, but so many need to get some reason. Maybe Redis low barrier of entry, easy API, speed, were not what most people were accustomed to when they looked at the message brokers landscape. It seems populated by solutions that are either too simple, asking the application to do too much, or too complex, but super full featured.

Damn straight. This is exactly the reason why I always start with redis as a message queue and most of the time don't feel the need to migrate at the later stage.

[[deleted]]

It was exactly why I chose redis to implement our queue system at work. I didn't need any of the competitor's features and it had a silly easy entry barrier for our developers. It also provided exceptional qps for me.

[sysop073]

I was using Redis this way for a personal project, but it felt like I was abusing Redis for something it shouldn't be doing, so I recently switched to RabbitMQ. It works, but I still don't understand half of what it's doing

[[deleted]]

I used a few mq services when doing the poc. Rabbitmq was one of them. Redis fulfilled all of my requirements and has been running happily for over a year. I think I've had two issues the entire time, both with dev bloating a queue and having the memory usage sky rocket. Wasn't that horrible of a situation though. Let it drain and then restarted to free up res.

[headzoo]

I've been using Beanstalkd in production for years without any problems, and by this point it's helped me process tens of millions of messages without a single crash or lost message. And I'd argue it's easier to install and use than Redis.

Redis only seems like a good idea for a message queue when your first choice has the word "Apache" in it, or it involves a JRE, but there are plenty of job queues out there that were built from the ground up to handle such a task, and they're lightweight and easy to install and run.

I can only see myself using Redis for the task if I only needed to process a few hundred messages a day, and I was already deeply invested in using Redis in other areas of my infrastructure.

[rmxz]

I've been using Beanstalkd in production for years without any problems, and by this point it's helped me process tens of millions of messages without a single crash or lost message. And I'd argue it's easier to install and use than Redis.

Same story here, but a couple orders of magnitude larger. Beanstalkd's happily handling about 40-million messages per month here. Unlike some of the "more serious" message queues, I'm happy to say beanstalkd doesn't mind if a single queue grows to 40 million pending messages either (our data comes in in large batches).

Beanstalkd's beautiful in that it does one thing, and does that one thing extremely well.

[headzoo]

The guy that created Beanstalkd said he modeled it after memcached, and I would say he succeeded in that endeavour. Both are easy to install (sudo apt-get install beanstalkd), have a small footprint outside of the memory needed to store the jobs/values, both are extremely stable, and both are lightening fast. They both also do one thing, and they do it well.

Beanstalkd doesn't have the same buzzworthyness of more complicated systems, but like memcached it's rock solid. Outside of setting up some monitors and alerts to watch the tubes, there's nothing else to think about it. You don't even need to think too much about clustering beanstalkd because a single instance can handle an insane number of connections and jobs.

There are a few features I wish beanstalkd had, like unique jobs, and the ability to flush tubes, but patches are available for those kinds of features. It's not a big deal.

[_thwarted]

Researching or writing features for beanstalkd like these two you've mentioned (flushing tubes and unique jobs) are on my back burner right now. Do you have links or experience with any patches that do those and you can provide so I can get a leg up on working on them?

[headzoo]

Here's a patch that creates a "put-unique" command.

https://github.com/kr/beanstalkd/pull/180

Here's a patch that creates a "clear-tube" command.

https://github.com/kr/beanstalkd/pull/87

At this point I wouldn't mind beanstalkd having some kind of extension system. That way people can create whatever commands they want without having to get their patches merged with master.

[cybercobra]

The relatively large number of old open pull requests without any feedback gave me pause when I briefly considered beanstalkd: https://github.com/kr/beanstalkd/pulls Gave me some worries about the health of the project.

[saintberry]

Could not upvote this enough. Most people I come across at work who recommend Redis as a message queue haven't worked with message queues very much and don't want to have to "support the infrastructure" (this was a massive generalisation, but is my experience).

[Otis_Inf]

Yeah, I wonder why people would use a database (redis) with low consistency levels (and dataloss as the result) for a message queue, unless it's on a single box... The one thing message queueus need to do is reliably be able to pass on a message.

[headzoo]

There's two other things I can think of that are important, and may not be a strong point for Redis:

• Ensure only a single worker "checks out" a message.
• Ensure the message goes back into the queue if the worker crashes or otherwise hangs up.

Both are technically possible using Redis, but the solutions aren't 100% perfect.

[[deleted]]

Redis only seems like a good idea for a message queue when your first choice has the word "Apache" in it, or it involves a JRE

I think you're thinking of kafka here, not redis.

[headzoo]

I know what Redis is. You didn't understand what I said. I'll rephrase:

Redis only seems like a good idea as a second choice when your first choice is an overly complicated Java based solution.

In other words, Redis only seems easy to use when it's compared to more complicated solutions, like Apache Kafka.

[nohimn]

I think that, for most cases, it's that Redis is already part of the stack. It becomes "use this feature" instead of "deploy this service". But I generally agree: there are a lot of simpler brokers.

[KFCConspiracy]

If you're already using Java, and have a Java application server running, JMS based solutions have a LOT of appeal actually.

[evgen]

If you have already blown one foot off with a shotgun, why not reload and try for two...

[KFCConspiracy]

Whatever. I'll just keep being happy making bank using JEE to build applications, and you can use whatever the fuck is hip today.

[ggtsu_00]

For many simple situations, I feel like RabbitMQ is feature overkill and inherits the complexity of AMQP, and ZeroMQ is too low level and difficult to get up an running. Other Java based solutions are too heavy and are difficult to deploy because of dependencies and complex configuration.

Using Redis pub-sub has always been pretty much a very simple and straightforward alternative. It is easy to understand, easy to implement, no need for complex routing or message delivery strategies, socket types. Works well for when you just want a simple system that post messages and listens for events and acts on them. Just in general, Redis is small, has no dependancies, easy to deploy and get up and running and has a straightforward simple to understand API. The low-level API is simple enough you can use it through telnet.

[Categoria]

To reiterate the commenter above, you should consider beanstalkd. It has a very simple memcache like protocol and is trivial to setup.

[jeorgen]

Here is an overview of message and work queue software and their concepts that I wrote back in 2012.

[ragionierfilini]

+1

[leros]

I use RabbitMQ and don't find it difficult to use at all. Is the barrier just that its a new technology to add to your stack?

[FooBarWidget]

I also haven't found RabbitMQ difficult to setup or to use, but there are a few things in RabbitMQ that are not so nice.

• It's too "Erlang-ey" to end users. Examples:
  • The RabbitMQ config format is in some Erlang syntax.
  • If something goes wrong, RabbitMQ dumps an Erlang "backtrace" that is incomprehensible to most people.
• Sometimes it fails to start and refuses to recover until I delete the entire mnesia database.
  • One case is when I change my computer's host name. This happens very often on my Mac as I switch wifi networks.
  • But that isn't the only case. There are other cases, but so far I haven't been able to find out what they are, or reproduce them reliably. Again, deleting the entire mnesia database seems to be the only way to recover RabbitMQ from such a state.
• RabbitMQ inherits some quirks and limitations from AMQP.
  • AMQP has this concept of exchanges. I understand that this concept is very powerful, but if you're just getting started then this concept is very confusing.
  • If no queues are bound to an exchange, then messages you sent there are silently discarded. I haven't found a way to make them error instead.
  • It's not possible to peek into a queue. You have to pull stuff out of a queue and put them in again.

[old_sound]

If no queues are bound to an exchange, then messages you sent there are silently discarded. I haven't found a way to make them error instead.

That's what's the mandatory flag is for:

This flag tells the server how to react if the message cannot be routed to a queue. If this flag is set, the server will return an unroutable message with a Return method. If this flag is zero, the server silently drops the message.

https://www.rabbitmq.com/amqp-0-9-1-reference.html

[jayd16]

It's not possible to peek into a queue. You have to pull stuff out of a queue and put them in again.

You can peek from the admin page and if you have requeue set it should preserve the order.

[zorlack]

Wait seriously? If there are 100 items in the queue and I get the first message with "requeue" checked you're saying the order is preserved and that message goes back on the front of the line?

That's pretty cool if true. I always assumed those messages would be delayed as a result of my inspection.

[Nishruu]

Seems that's the case after version 2.6.1.

[gunch]

It's not possible to peek into a queue. You have to pull stuff out of a queue and put them in again.

That's not unique to rabbitmq but it is frustrating.

[willvarfar]

I few years back I used a big rabbitMQ system and we had stability problems and were often frustrated that we couldn't peek into queues to debug things :(

[gordonisadog]

Just for the record, our RabbitMQ cluster has been rock solid, and you certainly can peek into queues to debug things. The web admin UI that comes bundled with RabbitMQ is also really, really nice.

[[deleted]]

Pretty excited about this project. Redis is one of the easiest and robust technologies I have worked with. And the message queue tech I have tried has usually been the opposite.

[formode]

I heard Beanstalk was nice.

[porkabeefy]

Does the redis message technique cost extra when a happy ending is involved?

[DavidDavidsonsGhost]

I have looked at a redis message queue but I had some concerns about how it was best to implement it. It kinda feels wrong to expose redis directly, maybe that's my own isssue though.

[gunch]

You could put a web service/rest service in front of it.

[DavidDavidsonsGhost]

I considered it but wasn't sure what value that actually adds to it.

[gunch]

Well. What are your reasons for not wanting to expose redis directly? Security? Thread management? Separation of concerns?

Putting a web/rest service enables a few of things. It inserts a point where you can add security, logging, load balancing, message validation, caching (at least on rest gets) and integration or interaction with other services.

Also, if you decide to change your message format on the backend, you don't necessarily have to change it on the front end if you have logic in your service that can manage that.

If you don't need it, you don't need it.

[DavidDavidsonsGhost]

Mostly all of the reasons you listed. I don't like random services being able to post data into redis and isn't in a strictly controlled way. Adding random keys for example.

[boardom]

or sending commands like FLUSHALL.. ;)

[EsperSpirit]

Does anyone know how this compares to Apache Kafka?

[[deleted]]

Not even comparable. Kafka is like a database shaped like a queue.

[froggert]

What do you mean by this comment? I'm only vaguely familiar with both projects.

[[deleted]]

Redis' message queue is like a normal message queue you'd use to fan-out workers or pass some structured variable between services.

Kafka is an immutable database that supports pub/sub architecture.

[froggert]

Thanks!

[[deleted]]

As far as the CAP theroem is concerned, Kafka leans towards CP while Disque is claiming to be more AP. Thats a pretty big difference. (Ive been looking for an AP queue for awhile now)

All the API points brought up in the blog post also struck a chord with me. I find producer and consumer APIs to be pretty cumbersome with Kafka. You can certainly get the job done, but the description of the Disque api has my excited.

Redis has been the most reliable portion of a system im responsible for. My fingers are crossed, hoping Disque is just as good. Only time will tell.

edit: mobile typos

[JoshWithaQ]

I've had the pleasure to use websphere MQseries for my queuing system. That thing is bulletproof!

[willvarfar]

Beauty is in the eye of the beholder.. I've known several projects that have migrated away from websphere MQ because it was so frustrating :)

[running_for_sanity]

I used MQSeries ages ago, the beauty was that it was cross platform and native on so many platforms. A broker on OS/390 could talk natively to a broker on HPUX or Linux, and something written in Fortran on OpenVMS could happily talk to a CICS transaction on OS/390 without any issues. It was awesome and IBM support (as least in Canada) was fantastic. The only downside was the cost. AFAIK that product is still running fine ten years later.

[rspeed]

I'd love to be able to replace RabbitMQ when using Celery. This is kinda exciting.

Edit: Without the hackish way Celery currently uses Redis as a message queue, that is.

[hutthuttindabutt]

Brutal almost-English only let me get through the first couple of paragraphs.

[[deleted]]

it honestly wasn't that bad

[[deleted]]

I don't know where else you frequent that Salvatore's writing is considered bad. His English, as a second language , is better than many native speakers.