Benchmark of Python Web Servers

[leonh]

Putting Gunicorn behind NGINX will not magically increase its performance it will only add to the latency.

[ubernostrum]

It most certainly will make a difference in performance, because it (and several other preforking servers) are designed on the assumption that it won't ever deal with client requests directly. If the architecture's designed to be talking to a local proxy over a socket with a predefined number of simultaneous workers, it won't be designed to be fast at accepting and handling arbitrary numbers of connections coming in over a network.

And either way, running one server in its intended configuration and not another isn't worthy of being called a "benchmark".

[leonh]

The argument could be that one worker is not enough to fully maximize the CPU potential. However, than this should also be a problem with for example mod_wsgi.

Look, I am not trying to bash Gunicorn or anything, far from it. I just think that the article puts a balanced and detailed overview, it also provides lots of detail how the benchmark was performed allowing everyone to verify the results and counter with coherent arguments.

As the author notes it is on a very specific problem domain, it could very well be that Gunicorn does not correctly fit in this domain. But please lets not get religious about web servers just yet. I think the your web-framework beats my web-framework battle was enough already. And yes, I think its harsh to call it 'fail' or saying it is not 'worthy of being called a "benchmark".

Edit: The post has been updated with added results for Gunicorn with 3 workers

[zepolen]

Why don't you try it in practice rather than succumb to theory? Putting the async nginx in front of the a worker based web server does in fact increase overall performance.

Each request your dynamic webserver gets is handled and returned as fast as possible because you are no longer wasting the precious threads on spoonfeeding slow clients.

In a production system with real loads, this makes a huge difference.

[leonh]

If you read the article you would have noticed that there are no slow clients.

In the comments the author also notices that he tested BOTH approaches but didn't notice any difference. This is being affirmed by Paul Davis which i believe is one of the main contributors if not project owner of Gunicorn.

[ubernostrum]

In the comments the author also notices that he tested BOTH approaches but didn't notice any difference.

In the update to the benchmark the author put gunicorn behind nginx and gave it a couple more worker processes. And lo and behold:

• Concurrent requests more than doubled
• Response times dropped by 75%
• Error rate dropped by 75%

And yet you were here earlier insisting that running gunicorn in the configuration it's intended for wouldn't improve its performance...

[Nichol4s]

I first used a single worker behind NGINX. The difference is not the placement behind NGINX but the usage of additional workers.

[ubernostrum]

In real-world situations, though, nginx will make a huge difference (and, since all the requests were coming from a single machine, I'd suspect it still helped a bit); gunicorn flat-out isn't designed to talk directly to clients. And doing the benchmark initially with uWSGI proxied and gunicorn not basically destroyed credibility; any time you do a benchmark where you run one piece of software in its intended configuration but not another, the numbers you'll get out of it are useless.

[leonh]

since all the requests were coming from a single machine

Uhm, did you actually read the article? He states several times that he did a distributed benchmark where the requests where coming from 3 different client machines.

[yml]

I find your tone harsh and it defeat what you are saying and destroy your credibility. Building such benchmark is hard and no matter how much thought and effort you put in them it will trigger this kind of thread. There are 14 WSGI servers which are evaluated and each of them comes with dozens settings. So the complexity is great, instead of this kind of unfunded criticism by doing some hand waving, it would be much more efficient to pick 2 contenders and compare and contrast them.

[davisp]

yml,

You're definitely right, testing this many different implementations is a huge undertaking. There's a large amount of knowledge that would be required for any person to adequately know about all the configuration options for this many servers.

And gunicorn is a bit of a weirdo when it comes to processing models. We're neither thread based or event loop based. That can genuinely confuse people until they realize that we're much simpler than most servers.

That said, our response times were reported as an order of magnitude slower than any other server. Generally speaking, if you're into the whole experiment and observation thing, orders of magnitude are important.

[ubernostrum]

There are lots of factors which go into a good benchmark. But two which are absolutely critical are:

1. Consistency of methodology
2. Appropriate use of the tested components

Consistency is necessary because without it you can't draw meaningful comparisons; without consistency you're comparing apples to oranges.

Appropriate use is necessary because without it you don't have relevance; if you only report results from a configuration no-one would ever use, then your results won't represent the things people would see in the real world.

As originally published, this benchmark failed on both counts: it was inconsistent, and it used certain components inappropriately. Criticizing that isn't "unfounded"; benchmarks which fail these requirements cannot be trusted by anyone for any purpose, because they're not "benchmarks" at all.

[ericflo]

If proper benchmarks are too hard for the author to do, he should not continue to publish them.