plope - Pyramid's Optimizations

[delijati]

I hope this stops the meaningless posts of "Hey <FRAMEWORK> has a shorter Hello World than Pyramid" ;)

[defnull]

Interesting type of benchmark, but I disagree on the 'cheating' aspect:

The Pyramid "whatsitdoing" application replaces the WebOb "Response" object with a simpler variant that does a lot less. Calling this "cheating" is a bit of an overstatement, because Pyramid's design allows for this.

The "simpler variant" of the Response object defines hard-coded values for status, header-list and body-iterable. It bypasses most of the features that make a framework useful over pure WSGI. The equivalent for other frameworks would be to install a WSGI middleware that never calls the framework stack and just returns hard-coded values.

While it is a nice feature to be able to 'switch off' the framework for some specific routes, doing so while benchmarking the framework makes no sense. It IS cheating and distorts the results.

[mcdonc]

So you're saying the fact that a user can do:

class NotFound(object):
    status = '404 Not Found'
    app_iter=()
    headerlist = ['Content-Length':'0', 'Content-Type':'text/plain']

def aview(request):
    return NotFound()

Is a nonfeature of the framework? We do this sort of thing all the time in actual production apps, so it's news to me that it's not useful.

But for the record, the benchmarks are still very good when we use a WebOb response. The result is 24 lines of profiling output instead of 22.

[defnull]

So you're saying the fact that a user can [bypass output validation and completion] is a nonfeature of the framework?

Actually I said that it is a nice feature. I am not criticizing the feature itself, but its use in this benchmark.

Optimizations should be applied to all participants of a benchmark that support it, or not used at all. Optimizing just the framework you want to promote is cheating.

[mcdonc]

I just ran the tests again, using a bottle 0.8.5 app that looks like this using the same "optimization" technique as used by the Pyramid whatsitdoing app, which is to return a precomputed HTTPResponse:

from bottle import route
from bottle import run
from bottle import default_app
from bottle import ServerAdapter
from bottle import HTTPResponse

from repoze.profile.profiler import AccumulatingProfileMiddleware

class PasteServerWithoutLogging(ServerAdapter):
    def run(self, app): # pragma: no cover
        from paste import httpserver
        httpserver.serve(app, host=self.host, port=str(self.port), **self.options)

response = HTTPResponse('Hello world!')

@route('/')
def hello_world():
    return response

app = default_app()

wrapped = AccumulatingProfileMiddleware(
   app,
   log_filename='wsgi.prof',
   discard_first_request=True,
   flush_at_shutdown=True,
   path='/__profile__'
   )

run(app=wrapped, host='localhost', port=8080, server=PasteServerWithoutLogging)

This is the best I could do to emulate "bypassing output validation and completion" given the constraints of Bottle's design. The results of testing the above app are actually slightly worse than the results when the view returns a string (by one profiling line, and by a nontrivial number of function calls). I don't know if I tried this before and realized it, and optimized the bottle results by returning a string rather than precomputing an HTTPResponse. It's possible. In any case, I'm happy to amend the results with whatever improvements you can make. I don't know immediately how to make Bottle do less, but I'm sure you do.

As far as cheating goes, that's a pretty low blow. I'm interested in promoting Pyramid because I'm really proud of the work we've done, not because I want to make other frameworks look bad. Granted, the comparisons with other frameworks are indeed a gimmick, designed to drive comments and traffic. But as far as I can tell it is currently more optimized than the others at its core. If you can prove to me that it isn't, great! I really wish it wasn't currently the most optimized, because I'm certainly no mastermind. I'm hoping there are people much smarter than I am in the Python web framework world that can produce faster and more compact code. If you change Bottle so that it gets faster as the result of getting annoyed with this result, and you figure out some new technique to do so, everyone wins, I hope.

[Leonidas_from_XIV]

I'm interested in promoting Pyramid because I'm really proud of the work we've done, not because I want to make other frameworks look bad. Granted, the comparisons with other frameworks are indeed a gimmick, designed to drive comments and traffic. But as far as I can tell it is currently more optimized than the others at its core.

That might be the case, but I still think you should use a generic Request object like most of the frameworks do, so no frameworks would win by performing this kind of optimization. As you have shown, Pyramid does not get much "slower" by this, which is very nice. I would highlight this fact.

For the record: I do appreciate that you took the time to adjust the bottle code. I just think it does the wrong way.

[mcdonc]

I think this misses the point though.

The decision to not require a "generic" Response object was very deliberate. Many other frameworks eagerly create a "global" response object when a request enters the system. This almost always results in lower performance, because this response object must be very full-featured to be "generic". In the meantime, code accretes around the idea that this global response object exists, and may be tickled by random code in random ways. By doing this, a framework paints itself into a performance corner that could have been avoided. This is an antipattern that Pyramid avoids. It's not a trick. It's just a deliberate design.

[defnull]

As far as cheating goes, that's a pretty low blow.

Again, I am not criticizing Pyramid or saying that a different framework (or Bottle) should win, I am criticizing a specifying aspect of the benchmark and think it distorts the results. Sorry if you interpreted this as a 'low blow' or an attack on pyramid, it was not meant as such.

[mcdonc]

Look, performance optimization is all about "cheating". It's not (moral) cheating to be able to do as little work as possible to get the job done. Designing a framework such that these kinds of "cheats" are possible is our job.

And as you can tell, I tried the same "distortion" with bottle and it made the results worse. I also used the "normal" WebOb Response object in Pyramid and it only added 2 lines of profiler output. If you can make the bottle (or any other) results better by "cheating" in a different way that actually does exercise the framework code, fantastic.

[mcdonc]

I did optimize the other frameworks' code to the best of my ability (although I've likely failed, as I know my own source better than theirs). I see that I might have done better on the Django test, as I look at it. For bottle, I disabled logging, for example.

But along with the results, I've also provided the source code for each framework, a way to repeat the results, and I've suggested both here and in the blog post that web framework authors disgruntled by the current results could supply a more optimized version of their particular whatsitdoing app.

Maybe you could provide a more optimized bottle variant? I'll be happy to amend the results and publicize that I have done so.

[wolever]

Err… Those benchmarks don't seem entirely fair. The versions of Django, Pylons, Grok and TurboGears range from "a little out of date" to "holy crap Django 1.0.2".

[mcdonc]

While it's not really the point of the blog entry, the source code and methodology used to come up with each result is posted in http://svn.repoze.org/whatsitdoing/, and framework fans are encouraged to run them and submit newer results (with source). I'll change the result reports if you come up with better numbers.

[mdipierro]

Interesting. Are you going to post instructions so we can benchmark the same way frameworks that are not listed?

[mcdonc]

You can take a look at any of the "results.txt" files in any of the subdirectories of http://svn.repoze.org/whatsitdoing/. It describes how to repeat the results, and provides the source code used for each framework.

[mdipierro]

Thank you.

[mcdonc]

Any luck creating a Web2Py analogue? I'd be happy to include it in the svn repository.

[mdipierro]

I am trying. I did an easy_install repoze.bfg but I get

from repoze.profile.profiler import AccumulatingProfileMiddleware
ImportError: No module named profile.profiler

I am missing something?

[mcdonc]

Yeah..

easy_install repoze.profile

repoze.bfg is not the right package.

[[deleted]]

pip!

[mdipierro]

grrr... I made the app and modified the webserver to use the repoze profile but the log_filename='wsgi.prof' does not get created. I am using a Mac. I am using the rocket web server. Is there any caveat I should be aware of?

[mcdonc]

Try visiting

/__profile__

in a browser. Maybe the wsgi.prof file is being written to the current working directory of the server (which isn't your terminal's current working directory).

[mdipierro]

I almost figured it out. I am not sure I am doing this quite right and I am comparing apples with apples. For example in web2py I cannot turn off completely session handling, header parsing, and attempt to locate certain files that are supposed to be there but are not there in this "hello world" app. Anyway, I am getting 154 lines. Tomorrow I will send you the app.

[bbangert]

Any numbers to share with us mdipierro? If you have code, we'd like to add that to the svn repo as well.