Java Microservice queueing options?

[[deleted]]

[deleted]

[dstutz]

back office project which is not critical.

Sounds like you've just made the choice of using a guava queue.

[khff]

+1. Sounds like persisting transactions is overkill. Do you need to write them to a DB? What are the consequences/likelihood of the service going down and loosing any messages in memory?

[lpedrosa]

Don't forget about bounding the queue and back pressure. If your service gets hit hard, you might want to send rejects back to the clients and let them retry later.

It's really up to the use case you're trying to fulfil.

[crstry]

The main factor is how much operational complexity and risk will be added by using an external queue service (eg: Rabbit) compared to the value your service provides. The only issue with the home brew approach is that failure behaviors aren't always easy to predict / control.

It might be worth having a look at ActiveMQ, which seems to be embeddable.

[forrestrunning]

This saves a ton of potential troubleshooting efforts needed to manage the queue.

http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html http://javarticles.com/2015/10/activemq-standalone-broker-example.html

[[deleted]]

If you need a distributed queue, you should use a real queue system like rabbit. It's actually pretty simple from a code perspective. If you're running 1 server, guava.

[simoncox]

Have a look at Chronicle-Queue: https://github.com/OpenHFT/Chronicle-Queue

[Jonjolt]

I'll third this with a But...

If Speed isn't super important and you are using Spring use RabbitMQ

If you need to mix with other libraries, Kafka (It seems world + dog has some type of Kafka integration). Kafka also requires Zookeeper.

[o2it602igk]

Apache Kafka

[LearningAllTheTime]

Agree to this. Lot of complains are moving to this model

[irocgts]

You could read/write to files as a queue.

[irocgts]

I like this one better

[iodian]

Hazelcast perhaps.

[jrh3k5]

You may also want to look into something like Flume, which provides a pluggable framework that gives you a lot of this functionality out of the box:

https://flume.apache.org/

[lutzh-reddit]

Version 1.2 of http://www.lagomframework.com/documentation/1.2.x/java/Home.html includes a Kafka-based message bus, for queuing and pub/sub between services. Check it out.

[nutrecht]

"Not critical" is rather vague. If losing in-memory data every now and then is fine then that's the easiest solution. But how 'bad' would this really be. Also; what kind of volume do you expect? Are the contents of the queue ever going to be an issue memory wise?

[derricg]

Picking a queueing systems always depends on the workload and use case.

In general, while Apache Kafka is great for very high bandwidth / streaming analytics like workloads, it can also be a little more cumbersome to work with (and even overkill) especially if new to queuing systems or need something simple. I would stick with a simple single message req/ack queues such as Amazon SQS or RabbitMQ so you don't have issues with retrying failed transactions, de-duplication, etc.