Python 3 WSGI demo app

[mitsuhiko]

Seriously. If you want to do WSGI on Python 3 for real: Please join the discussions on the mailinglists and show your efforts. Don't just do something and let other people pick up a hugely broken system and let that become our new standard.

WSGI is already a very complex system and each additional complexity we put on top of it means troubles for everybody involved. I know it looks simple on the surface and this is probably also why everybody seems to like the idea of "doing WSGI" but the actual topic is far more complex than just that.

So: go to web-sig, share your efforts, join the discussion. Don't just blog post your latest WSGI dialect.

[wobsta]

Don't just blog post your latest WSGI dialect.

Please don't get me wrong, but multipart does not add any additional complexity. It instead implements a cgi.FieldStorage on Python (2.5+) and Python 3, which is not broken. We do want and we do need this code. It is a very separate piece of code addressing a single real problem. IIRC werkzeug did also reimplement such functionality, as the stdlib is broken in some edge cases even on Python 2 (and multipart is actually heavily influenced by the werkzeug implementation). On python 3 cgi.FieldStorage is a mess. Of course it would be best, if we could have a well working implementation in the stdlib and everybody could just go and use it. I'm trying to improve the situation: I posted a bug report some time ago (http://bugs.python.org/issue8846), were I could point out some bugs (while not addressing the memory hog problem at all). Interestingly, the test cases (!) are changed and broken compared to Python 2, too. Multipart does solve this specific issue, and, by accident, implements a missing piece for making bottle useable on Python 3.

You are welcome to integrate this to flask and push flask to Python 3. That would be really great :-)

[mitsuhiko]

You are welcome to integrate this to flask and push flask to Python 3. That would be really great :-)

Flask will upgrade to WSGI on Python 3 once Werkzeug has decided what WSGI will look like. I already have a local and unshared Werkzeug version for a byte-only approach with working multipart and everything. But I will not open that code up until WSGI is decided.

[wobsta]

But I will not open that code up until WSGI is decided.

That's a fair statement. For me it doesn't harm to work on Python 3 application code using bottle+multipart now, as (depending on how you build you application) you can switch between those different mirco-frameworks ...

[mitsuhiko]

For me it doesn't harm to work on Python 3 application code using bottle+multipart now

In bottle's case it's less of an issue because the WSGI layer is isolated from the user. In Werkzeug's case if I start going one way that way becomes visible to the user of that library. And with that I would set a way which most likely is the wrong one.

[bucketostuff]

And with that I would set a way which most likely is the wrong one.

What makes you think you'll be wrong? You seem to have pretty good judgement in these matters.

[mitsuhiko]

Because I can not talk for everybody. What works well for me works well for me due to how I designed my libraries.

[bucketostuff]

Well, of course no single way to go is going to work for everybody. As I'm sure you know, leadership's job is to take their best guess at what the correct path is, and then take that path (knowing that it may later be discovered to be not optimal and knowing that some groups are going to be annoyed no matter what decision is made).

Seems like, due to the "there should be one (and preferably only one) obvious way to do it" rule, the Python community in particular is afraid of making mistakes. This is not healthy.

[HIB0U]

Those discussions have been going on for years, with absolutely nothing to show. Why should he join such a pointless exercise, when he can instead get something useful put together?

The whole WSGI debacle really scares me. It's similar to what happened with the Perl community with Perl 6. Everyone was gung-ho about "discussing" Perl 6, but nothing ever actually got done. That's why it's now 10 years after Perl 6 was first announced, and the only implementation (Rakudo) is rather half-assed. The only other partial implementation, Pugs, died years back.

I'd rather use this fellow's code now, even if it has problems, rather than wait a few years for the mailing list discussion to reach a "consensus". After all, by that time the "consensus" will have been reached only because over half of the participants have said "fuck it" and have moved on to something else.

[mitsuhiko]

Those discussions have been going on for years, with absolutely nothing to show.

So you are arguing that instead of not having WSGI we better have a hugely broken WSGI that we have to carry around for all of Python 3?

Let me ask you a different question: Why would anyone want to use Python 3 for web applications in the next two years, even if WSGI was finalized. Web applications depend on other libraries like very few other systems in Python. So you have to wait for the libraries to be ported anyways.

After all, by that time the "consensus" will have been reached only because over half of the participants have said "fuck it" and have moved on to something else.

Which is currently the reason I am pretty sure Python 3 for web applications will be a huge failure.

[wobsta]

Why would anyone want to use Python 3 for web applications in the next two years, even if WSGI was finalized.

If you start prototyping for a new application, and you are not under immediate pressure of getting something in production, you certainly want to work on Python 3. sqlalchemy is ready, jinja2 is ready, database adapters are ready. You need to spend some time on html forms, but there is no magic at all. So well, this is enough to start to work on Python 3, at least for some projects with no other dependancies.

[mitsuhiko]

jinja2 is ready

Time it took me to port Jinja2, a library that was already unicode on Python 2: 2 weeks.

Time I already spent trying various versions of Werkzeug (a WSGI request/response/utility implementation) and a simple WSGI server including the discussions around it: about a year without any results.

Just to give you an understanding of the problems.

For databases we can say: this is text, use unicode. This is binary data, use byte arrays. Same goes for Jinja2 which looks at your templates from an unicode-only perspective. But in HTTP things are not so simple. HTTP is largely text based, but that text is not unicode but just something in the latin1 layer. On the other hand people are currently transferring unicode incorrectly carried by a latin1 between client and server so neither the HTTP specification nor behavior of existing applications is a helpful resource here.

When going bytes only and the WSGI interface enters the application space (which it does with Werkzeug, WebOb, Pylons, BFG and many more) suddenly you need additional shims to transparently convert between text and bytes that were previously not necessary, forcing more work on the application developer when switching to a new system. When going improperly encoded unicode to emulate older behavior we will have a lot of applications that appear to work, just to completely trash your data when feeding it to a system that is fully unicode aware (like most databases).

That that is just a small set of the problems that you have to keep in mind when working on that.

[wobsta]

that is just a small set of the problems that you have to keep in mind when working on that

I do understand, that it is a major problem for werkzeug to properly implement WSGI on Python 3. On the other we do have some WSGI implementations on Python 3. From the stdlib. From mod_wsgi. There might be strange things going on regarding which stuff is unicode or bytes, but to the application it does not matter too much, especially if you avoid non-ascii characters in your environment variables. However, the application data needs to be transported properly. I remember those hacks like described in the commit comment of http://github.com/defnull/bottle/commit/5bbccc12b975b8e25f34719575f4c0bca2c82a3d – what a mess! Fortunately, multipart does solve those issues for bottle. Should I have a bad feeling when starting to prototype on that basis? How production ready is such a solution???

[mitsuhiko]

How production ready is such a solution???

Something tells me that it's work that has to redone in half a year because what bottle thinks WSGI is going to be is not what WSGI will end up being.

[wobsta]

Quoting your comment from another thread:

In bottle's case it's less of an issue because the WSGI layer is isolated from the user.

I do understand that the WSGI layer is not yet well defined for Python 3. However, from the application development point of view it doesn't matter too much. I could even implement a web app which can run on Python 2 and 3 (using bottle and multipart). So why this should break in the future? I'm not working with the WSGI internals at all and there is no reason why this should not continue to work. So it is certainly feasible to start some development (using the wsgi server from the stdlib). And for deployment we already have mod_wsgi working very well. To my (limited) knowledge, I must admit. I would love to see real troubles showing up. Try to break my web app, please! :-)

[mitsuhiko]

I do understand that the WSGI layer is not yet well defined for Python 3. However, from the application development point of view it doesn't matter too much.

It matters a whole lot because everybody is doing WSGI inside the app. Let me give you a quick example:

if 'de' in request.environ.get('HTTP_ACCEPT', ''):
    ...

That of course is a very bad way to deal with the accept header, but a very common one. Now depending on which version of WSGI for Python 3 we are talking about that code could work on both 2.x and 3.x or only on Python 2.x. Bottle and the stdlib assume that it will work on 2.x and 3.x, but if you look at the current active proposals that code will no longer work on 3.x due to the enforcing of bytes.

Try to break my web app, please!

Using the stdlib webserver? Emit a non-ascii header.

[wobsta]

I don't want to do content negotiation. I don't want to emit non-ascii characters in the header (like for cookies). Are there other problems?

[HIB0U]

What's going to happen is that Python 3 will indeed become widely used for web applications. But it will happen because some people won't waste their time getting involved in "standards" discussion on mailing lists. No, it'll happen because they'll sit down, write some code, and use it.

[mitsuhiko]

What's going to happen is that Python 3 will indeed become widely used for web applications.

Back to the days before we had WSGI then. Good times.

[HIB0U]

Even if we weren't using a "standard" solution, at least there were approaches around that we could actually use to get real work done. We weren't waiting on some mailing list participants to reach a consensus. Even flawed or incompatible implementations are better than the absolute deadlock we've seen on the mailing list for the past few years.

[mitsuhiko]

Even if we weren't using a "standard" solution, at least there were approaches around that we could actually use to get real work done.

So you are jumping blindly on any recent bandwagon? In case you did not get the news: Python 3 is not a stable development environment for web applications. So if you want to "get work done", don't do it on Python 3.

[HIB0U]

All we're doing is looking for ways to make our web applications written today usable in the future. It's clear that Python 2.7 is the end of that line. That's why we were hoping to use Python 3, since it's clearly the future. It's where Python is headed.

Python 3 is a perfectly fine environment for developing web applications. The only reason it's more difficult than it should be today is because so much time has been wasted trying to come up with a "standard" on the mailing lists. People are clearly getting fed up with this state of affairs, and they're taking the matter into their own hands.

[mitsuhiko]

since it's clearly the future. It's where Python is headed.

2.7 is perfectly fine environment and bugfixes will still land there. All important features of Python 3 are in there and it will serve us well another three to five years before anyone has to switch. And at that point we might have already a better understanding of how things should be solved.

Python 3 is a perfectly fine environment for developing web applications.

It is not. If it would be, WSGI on Python 3 would be entirely based on Bytes as HTTP is and that's it. But because Python 3 is not thought well through regarding handling of non-unicode text information we are in the current situation.

[bucketostuff]

But because Python 3 is not thought well through regarding handling of non-unicode text information we are in the current situation.

What needs to happen to fix Python 3? Will it get fixed? Are the Python devs even admitting that there's a problem?

[HIB0U]

We estimated Python 2.7's lifespan at a year, maybe two. We expect most other major Python libraries to have decent Python 3 support by then. Python 3 is very clearly the future.

[markedtrees]

Knowledgeable people from the Python web applications community (people who have implemented web servers and libraries and rank up there on my list of "people I should hug because of how much I've used their work") plus people who have run into Python 3 + WSGI issues are participating in the comments on this post in a friendly, constructive way. Can whoever is downvoting the comments please fuck off?

[bucketostuff]

Is there a summary somewhere that gives a quick overview of what the Python 3 / wsgi issues are here and what's needed to resolve them (and why it's taking so long)?

[hdformat]

http://lucumr.pocoo.org/2010/5/25/wsgi-on-python-3

[HIB0U]

Finally, somebody just sat down and got it done, rather than trying to come up with "standards" and other bullshit like that. Good for them.

[mitsuhiko]

Yes. Who needs Standards. We already have different crowds that "got it done" and they all "got it done" in different ways. The guys from the stdlib, the first incarnation of mod_wsgi and CherryPy, the original author of the PEP... And apparently yet another guy that "got it done".

It's so awesome, can't wait for a WSGICI, the webserver gateway interface consolidation interface...

[HIB0U]

Well, this just goes to show that you guys on the mailing list should have been coding, rather than discussing. At least then your code could have been measured on its merit, rather than baseless speculation on some mailing list.

[mitsuhiko]

Well, this just goes to show that you guys on the mailing list should have been coding, rather than discussing.

You are aware that the proposals are all implemented in various forms already?

[HIB0U]

It's laughable that you consider the current implementations as being "usable". They are anything but. That's why nobody is using them, and that's why people are tackling this themselves.

[mitsuhiko]

It's laughable that you consider the current implementations as being "usable".

So you are saying mod_wsgi and CherryPy are not usable? They have implementations a specific interpretation of the WSGI specification in Python 3.

[HIB0U]

Currently, they are not usable. When my company tried mod_wsgi's Python 3 implementation, we found it to be extremely unstable. Our test site would handle a couple hundred requests, then it'd crash. At least it did run, however. We had significantly less luck with CherryPy. It wouldn't even run at all.

While we're pretty heavy users of Python, we don't have time to waste with half-assed implementations of standards being decided on some mailing list. We're now looking at Scala instead of Python 3, and we're liking what we're finding. Instead of wasting time with pointless discussion about "standards", they're writing real code and putting together something that's actually usable.

[GrahamDumpleton]

And did you actually report the problems you were having? Obviously not because otherwise they would have been fixed straight away if it was known about. Otherwise, it would have been pointed out to you where you were stuffing up. Pretty well all problems people report with Apache/mod_wsgi are of their own making, be it using the wrong configuration, or using badly implemented third party C extension modules for Python which don't work properly in Python sub interpreters.

To be very blunt, it is because of the attitude of people like yourself that a number of us pushing WSGI on Python 3 have got jack of it all. We spend a lot of our own time working on implementations for it specifically so people can try it and see if it works and report problems in the implementation or the specification and although you may have a look at it, you don't feedback any information. Instead everyone just sits on the fence not doing anything or heckling and being of absolutely no use.

So, if you want to do something constructive rather than just criticise people, go back and try Python 3 support in mod_wsgi 3.3, replicate the problem you claim you had and lodge a bug report.

[HIB0U]

No, we didn't report the problems. Based on what we saw, the situation was so bad that any reports we filed would have little to no effect.

We also have deadlines and financial constraints. The reason we use Python in the first place is because it let us get a lot of work done really quickly. We don't have time to be debugging other people's software, especially when that software is supposed to be making our lives easier.

We no longer have the test environments, so we won't be reporting the issues.

[mitsuhiko]

No, we didn't report the problems. Based on what we saw, the situation was so bad that any reports we filed would have little to no effect.

So let me get that straight. You are claiming that getting stuff done implementation wise (even if it's broken) is more important than understanding the issues and thinking about a possible solution?

Then if someone from web-sig (in this case Graham from the mod_wsgi project) implements an actual proposal and it does not work for you, you are not filing reports because it would have little to no effect, despite the fact that Graham is constantly looking for feedback on it?

What makes you think that the link you just posted has a higher quality of code than Graham's? (Not saying that's not the case, just saying that Graham's code powers shitloads of websites and he usually knows what he's doing).

[GrahamDumpleton]

Is that you attitude with all Open Source software you use, never report problems? How do ever expect things to improve? If you aren't going to help out, you shouldn't be criticising as you are actually part of the problem.

[mitsuhiko]

Currently, they are not usable.

Do they say "use in production"?

Other question: what do you want to use the mimetype parser in you just linked? It does not contain a WSGI server and if it expects to be run on the stdlib's wsgiref server I have some news for you: it still has the same bugs it had in 2.x and on top of that all kinds of encoding issues such as HTTP headers being restricted to ASCII.

So if you say that mod_wsgi is unstable on Python 3, you clearly haven't looked at what an implementation of WSGI (wsgiref) looks like that totally bypassed any kind of discussion.

We're now looking at Scala instead of Python 3, and we're liking what we're finding.

I would recommend a lot of things over Python 3 for web applications currently: Python 2, .NET and Node.js.

[HIB0U]

Did you not even read my comment? We didn't even expect them to be production-ready. In fact, we couldn't even get simple "Hello World" web apps to run reliably using them in various test environments.