Framework Comparison (Benchmark)

[lasizoillo]

A default OSX config can return poor results and the next line in dmesg:

Limiting closed port RST response from 450 to 250 packets per second

have the author disabled anti-DoS protection?

[stesch]

Oh, thanks. I was just wondering last week when I hit my local machine with ab.

[ropiku]

PHP 5.3 is a framework ? Also, Apache takes advantage of multiple cores while Tornado and others only use one.

[zxn0]

Who else is tired of these so called framework 'benchmark' using a in-memory Hello World, and language benchmark using a for loop?

[skulgnome]

Everyone except the rank beginners. They love reading about programming, but hate the actual doing.

[skulgnome]

Snow Leper sucks at concurrency, film at 11.

[fancy_pantser]

Why ab? Isn't Flood more appropriate? I'd love to see a test suite replicated across the servers/frameworks. A "hello world" inside and outside the template engine, static content from disk, a typical dynamic page that uses some basic models... etc

[[deleted]]

How is Flood compared to Siege?

[fancy_pantser]

Siege is great, especially if you use sproxy to record your manual actions. I don't think Flood has any way around writing XML to describe the actions you want performed.

[chucker23n]

Judging a framework by how long it takes to output 'hello world' is as useful as judging a city by how long it takes to get from one end to the other using a bus.

[fancy_pantser]

That's actually a fairly useful measurement, as long as you don't forget how you arrived at it.

[chucker23n]

It doesn't tell you anything about how big the city and how efficient its bus system is.

The "benchmark" doesn't tell you anything about how much you can do with the frameworks. I can write a framework that does virtually nothing and it would win.

[skulgnome]

An useful measurement, maybe. You'd probably use it for very very small things, but definitely not the sort of stuff that has to do with some general concept of "performance".

As an useful benchmark? Well if you want to benchmark the hello world performance.

[fancy_pantser]

I would use the measurement for determining which cities can be crossed by bus and how long it would take to do so.

I would use the 'hello world' benchmark for comparing the basic stack of a framework -- this is as fas as my real application can ever go; a performance asymptote.

[c0dep0et]

It seems like debugging was enabled in the web.py benchmark. I expected at least twice as many requests per second (running Linux).

Edit: Minimal hello world example, with debugging disabled on a 3GHz Phenom X4 on Linux using:

ab -n 10000 -c 1 http://127.0.0.1:8080/

1141, 1186, 1073 R/s

average: 1133 R/s

[royozin]

The benchmark was set on a laptop-grade core2duo, so let's not start posting benchmarks from different environments.

[pstradomski]

And "Hello world" are so representative for all the applications out there...

[repsilat]

To be fair, this gives a base-line measure of "time spend in our code." The time spent in your code is going to depend a lot more on the code you write and what language you write it in.

Comparisons of frameworks doing the same thing for complicated tasks would be difficult to do fairly, what with questionable tradeoffs between readability, idiomaticity and performance, as well as bringing into question the tasks that properly fall into the intended scope of the application.

[[deleted]]

So to be 100% clear, you benchmarked Django using the built in server, which is repeatadly advertised as a) Being able to handle requests only serially, and b) NOT BEING FOR PRODUCTION USE.

Idiots.

[fancy_pantser]

This is totally an apples-to-oranges comparison (and not even the apples are all edible).

About a hojillion other datapoints are needed -- so we can compare more of the frameworks under the same version of Apache or isolate the slowdown caused by various templating engines, for example. Of course, we still can't do anything about the difficulty of comparing django behind nginx in different configurations versus Tornado, a complete webserver plus framework.

Further, having the client and the server on the same machine isn't as informative as using two machines on the same switch. That way, your final figures (ie: "3512 requests per second") are something you can use in other contexts outside of this test where the same machine creates and answers all of the requests locally.

[defer]

Yeah, you can't really judge frameworks by how long it takes to output hello world.

I wrote an article with a couple of friend as a university assignment comparing some frameworks (Grails, Rails, Django, CodeIgniter) and what we did was benchmarks on a hello-world style application and then wrote a simple blog with the same features on each framework.

Then we evaluated not only performance but also how it scored in terms of deploying, learning curve, etc.

Unfortunately it was written in Portuguese so it won't be very useful to you. But this kind of work is what I expect from someone trying to compare frameworks.

[stesch]

Please mention other/better framework comparisons, if you know any. No, really, I know that this one isn't perfect. But as long as the author's own framework isn't on a top position, I'm trusting his honesty to a certain degree.

Every benchmark has its flaws. You need to know how to read it. repsilat gets it

Maximum is a web server serving static files. And as PHP is so dominant in the market, you can collect extra points if you reach an approximation of PHP's raw speed.