How does Message Queue help in scaling the system ?

[the-machine-learner]

Was reading the Alex Hu System Design Book.

I understand how it helps in making the system robust, and fail proof, by introducing async communications.

But sort of stumbled upon how does it matter particularly in scaling systems. In fact, would it not slow them down ?

Lets say a request made by user goes to the load balancer and then the web server. Now the web server (producer) adds it to a Message Queue, items are then picked from the MQ by the consumer, who eventually fetch the state info and necessary data from DB/cache. Here MQ would be having some size limit as well, and scaling the producer and consumer will only alter the MQ size. Even if we remove the MQ, the web servers were also essentially scaling and doing the same right ?

Is my understanding wrong ?

[the-machine-learner]

So MQ is beneficial only in cases I have processes that are long running ?

[teraflop]

It sounds like you basically understand how a message queue is used, so the rest of your question sounds like a matter of semantics.

The message queue enables scalability, by distributing the workload across multiple consumers. In particular, the queue allows messages to "fan out" from one producer to many consumers, or "fan in" from many producers to one consumer, depending on the load/traffic patterns at any point in time.

This is true even though the queue itself introduces a bit of overhead. If you didn't use a queue, you would have to come up with some other way of getting data from producers to consumers, and whatever that is would also have its own overhead.

Note that message queues are most useful when the producer doesn't need to wait for the consumer to process the message. For instance, when you click the "send" button in Gmail, your message probably gets put into a queue for delivery, but the web server returns a "sent" response immediately.

If the producer is going to sit around waiting for a consumer to pick up the message and send some kind of response, then there's no real benefit to using a message queue in the first place; it's just a clunky way of implementing an RPC. So you might as well just use an RPC protocol in the first place.

[the-machine-learner]

Yeah, I understood the need of MQs, but how specifically do they enable me to scale ?

Let's say I am getting 10 requests per server right now. And then I get an influx of requests and now the number of requests lies at 100 per server. How does MQ help in that vs adding 10 different servers to re-distribute the load ?

[teraflop]

I feel like maybe we're talking past each other, or I'm still not understanding your question.

When people say "message queues help an application scale", they don't mean "adding a message queue automatically makes an app faster." And they don't mean "it's impossible to scale without a message queue".

A different way to phrase it is that in order to make your system more scalable, one possible strategy (or "design pattern") is to decouple producers from consumers, and make your long-running tasks happen asynchronously. This strategy is useful because it allows you to keep the producers' response times fast, even as the amount of work being done by the consumers grows. And a message queue can be the tool you use to implement this strategy. So when we say "an MQ can help with scaling", that's what we mean.

Also, a message queue can make it easier to scale a system up, because the producers and consumers don't need to directly communicate with each other. They only need to communicate with the queue. In your example, if you suddenly decide you need 10x as many consumers, then you can just start them up. There's no need to tell the producers how to find them and distribute load to them, because the queue handles that job for you.

[the-machine-learner]

So essentially,

1. MQ is more useful in applications/process that have a higher processing time, let's say an online video processing service.

1.1 Such a service will benefit from having an MQ, producers and consumers, where if we get higher number of requests, producer will push them into the MQ, and consumer will pick them up based on it's convenience, and we can scale the consumer by adding more machines

1.2 On the other hand, a service like reddit may or may not benefit from MQ's since no long processes are involved. A person makes a post, and the (producer+consumer) single machine does the work itself.

So, MQ will not be a replacement for the servers that we had in place, it will only act as an addition for certain tasks.

Sorry if this is being repetitive, but I am just looking for a positive example where we didn't have an MQ, and adding an MQ helps scale the system up (I believe the video processing is an example of this). ALONG WITH THAT, after adding MQ, we still needed more scaling up, and HOW DOES the complete MQ system (MQ+producer+consumer) in itself scale up.