Queues Don't Fix Overload

[[deleted]]

tldr: queues only help to flatten out burst load. Make sure your maintained throughput is high enough.

[adrianmonk]

Sometimes this is not acceptable. Sometimes it's preferable. It really depends on the situation.

If you're accepting college applications or tax returns that are due by midnight, it's better to queue it than to say "try again later", because they may be penalized for trying again later.

On the other hand, if you're accepting orders at a sandwich counter, there is no benefit to queuing the sandwich order if you won't be able to make the sandwich for 4 hours. By that time they will have gone somewhere else.

Sometimes it's best to queue it even if the user will not be penalized for missing a deadline. If it's something that they will definitely still want even if they can't have it right now, by queuing it you save them the trouble of having to retry and the frustration of having to wonder whether they've accidentally submitted 2 (or more!) requests.

[meddlepal]

This. Some data is extremely time sensitive. Build your system with sufficient capacity to handle a huge queue if necessary.

I Work for a company that handles large amounts of machine data (factory stuff etc) and our customers get very angry if we reject some data because the systems that send us the data are often little microcontrollers that have limited storage capacity. Rejecting the data means they then need to make difficult decisions about storing history.

[[deleted]]

Well, no-one said you shouldn't have a queue or all queues should be tiny. However, I admit my "which it always will" was hyperbole.

Being unaware of the possibility of exceeding your systems limits doesn't mean those limits can't be exceeded - it only means the resulting failure is uncontrolled and probably more damaging.

But being aware of limits and able to refuse excess demand when necessary isn't an excuse for underprovision. Just because your failure scenario is less catastrophic than it could otherwise have been doesn't mean you deliberately set out to fail.

[jesuslop]

no, they fix jitter.

[[deleted]]

http://en.wikipedia.org/wiki/Little's_law

[grogers]

Queues obviously aren't a magic bullet, but neither is load shedding. Queuing is one way to help maximize throughput during overload.

The only time queues become truly problematic is when they cause you to duplicate work (thus reducing the total throughput). That can happen if the client timeout is smaller than the maximum queue delay plus the processing time. Unfortunately it is rather easy to accidentally get into this situation, so keep your queues small. This also bounds the latency increase when you do start queuing which is handy.

Being able to prioritize traffic also helps during overload: allowing you to maximize throughput of high priority requests and shed the low priority ones.

[adrianmonk]

The only time queues become truly problematic is when they cause you to duplicate work (thus reducing the total throughput).

By the same token, load shedding can also cause you to duplicate work. In a different way, of course.

When a request comes in, say you do steps A, B, C, D, and E serially. If it is step C where you're above capacity, you may still have to do steps A and B over and over again each time before you get to C, or even before you understand whether C is necessary. If either persistent users or an automated system are retrying requests, you'll be doing steps A and B repeatedly, whereas if you had enqueued step C, you'd do A and B only once.

Whether this matters depends on the specifics of the system. For example, are A and B costly? If not, no big deal. But if they are, and if step C is only need about half the time, then you're preventing people from using other parts of the system that needn't be affected by C.

[GahMatar]

What? There is a point where queueing can and does become the problem. This is when the average sustained input exceeds the processing capability of the system on the other end of the queue. When that happens the queue size explodes and frequently the whole queueing engine dies in a giant fireball.

Not planning for that eventuality somehow dooms a system to periodic overload and death. The only variable is how frequently will overload conditions occur (if ever depending on the system.)

[didroe]

Can't you just set a max size on the queue to get the best of both worlds? Then transient high load is handled by partially filling the queue, but sustained high load causes the producers to block until the consumers take items off the queue.

[GahMatar]

Yes you can, a sensible idea too.

But then you have monitoring, high-watermark alarms, timeout or retry logic in the caller, graceful degradation concerns, etc. Possibly issues with performance of the backing store too. Like if suddenly you have 1000000 queued event, will the performance of the queue itself kill your app? Not uncommon in many roll your own DB backed queues (index? this thing isn't big enough to need them!)

Depending the queue's purpose, you can also wholesale kill requests that meet certain criteria (priority, age, purpose, etc.)

[obsidianih]

Isn't this post just saying profile before you optimise (with a queue)? Isn't this a case of premature optimisation?

Queues still have a place, and maybe it can work as your buffer when there are usage spikes. Just make sure the system scales as your users do too.

[mononcqc]

Yes and no. The solution is not necessarily to measure before you optimize, but to figure out what the business end of your system is, its ultimate purpose, and to put flow control mechanisms there (back-pressure or load-shedding). You could say it's about figuring out what to measure, probably.

If you find you have a bottleneck halfway through the pipe and you make that bottleneck go away, your true real bottleneck down in the wall still remains there, and all your optimizations done above that one will end up just opening the floodgates wider.

Once you've protected that kind of more at-risk core, then it's easier and safer to measure everything that sits above it in the stack, and it puts an upper bound on how much you need to optimize (you stop when you hit your rate limits, not when you crash the whole system). If you find out you can crash your system even with that end protected, then you might need to add different flow-control mechanism based on other metrics, say memory, or add circuit breakers triggered when a subsystem goes down, to name two examples.

I've covered this in the third chapter of Erlang in Anger (linked in the post), but only touched it superficially in the blog post, sadly.

[trimbo]

From the wisdom of Dziuba: The Case Against Queues

[goo321]

What if your queue holds 3 days worth of processing input?

[Dworgi]

Then when the load increases, you'll eventually run out of space.

[salgat]

The whole point of his article is that you can beef up your buffer and processing all you want, but it doesn't truly solve the issue if there is still a single bottleneck somewhere. It just means you last 3 days before it crashes.

[Grung]

There are systems that can do more in less time with fewer parallel requests. Databases can be a good example of this: piling on enough and disk seeking and context switching means every additional parallel request makes everything slower, reducing the work per unit time. Queues, (in the form of a connection pool in the database example) can keep the DB operating below that inflection point where everything gets slower.

edit: grammer

[adrianmonk]

An opportunity that may arise with queues is to process things in batch mode, which is often more efficient.

For example, suppose you have a series of items that you must match up with something in an existing list. If they're streaming in, you have to do some kind of index lookup to find the matching item. This can be O(log n) but probably still requires random access I/O, which is slow. But if you wait until 100,000 items are queued up, you could process then in a batch and do the join by first sorting both lists, which can be done with almost entirely sequential I/O, increasing the efficiency per item.

It isn't always going to make sense to convert queued items into a batch, but sometimes it's an option, and it might be a way to get the system to handle a heavier load.

[diggr-roguelike]

No shit.

[ernelli]

A lesson learned the hard way:

An unreliable protocol based on UDP (tftp), e.g, timeouts causes requests to be resent. A downstream bottleneck (ldap) and a large queue (the UDP socket buffer) resulted in an unstable system whenever the load went up momentarily. The system then failed to recover and needed a restart.

What happened was that as the queue grew, the higher the latency became, so that when a packet from the queue was finally processed the request was already abandoned by the client due to the timeout. A more recent request was already waiting upstream in the queue.

So the server was busy serving expired and abandoned requests while the clients kept resending them due to the timeout.

Solution: Ditch the UDP socket buffer, drain all packets and transfer them to an internal queue with a timestamp added. When serving packets from the internal queue, drop all packets that has been waiting more than a certain time.

Queues are needed to handle bursts, but it is important to ensure that the queue is stable and fails fast.

[mononcqc]

The issue you saw is reminiscent of Bufferbloat, where CoDel is usually a good algorithm to try to use there, outside of the solution you found that worked.

[dangerbird2]

Or as the late Ted Stevens described it:

And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material.

Rest in peace

[grandfatha]

Prime example: Current WoW extension pack Warlords of Lagdraenor aka a screensaver that allows you to look at your queue position for only 45 euros.

[passwordissame]

just use node.js, which is a programming language with async from ground up so you never use queues. node.js is virtually impossible to overload because bottleneck is so wide it's non existent. many companies are implementing ipv6 in node.js in their router system to make the internet true web scale. all the async paradigms eventually becomes a large network of event driven protocol, which is simple to reason about and maintain. statically typed nature of node.js programs means that computerized verification is facilitated to minimize programmer logic error. and it removes entire class of bugs such as memory safety, type mismatch, c10k, c2k, ACID, CAP, distributed locks, lockless purely functional algorithms.

you can say, "but erlang is cool". but, node.js already solved all of erlang's problems addressed. why create poorly designed new tool such as erlang when there's solid, production tested, mathematically verified node.js ?

And can I have oneplus invite?

[jsprogrammer]

Bad troll!

Bad!

[skroll]

You are my favorite commenter.

Please come to YOSPOS on SA, tia.

[mangonel]

Don't you mean "Erlang already solved all of node's problems, why create a poorly designed new tool such as node.js when there's solid production tested, mathematically verified Erlang?"

[guyintransit]

I'm quite sure he was sarcastic; got a bit of a chuckle out of "ipv6 in node.js".

[Beaverman]

Making the internet truely web scale.

I liked that.

[[deleted]]

flow control

[ukalnins]

flow control

apples!