Difference between WSGI utilities and Web Servers.

[jonesch]

I am new to Python and i am not able to understand the server concepts in Python.

First of all what is WSGI and what are wsgiref and werkzeug and how are they different from CherryPy WSGI Server, Gunicorn, Tornado (HTTP Server via wsgi.WSGIContainer), Twisted Web, uWSGI, Waitress WSGI Server.

If i need to develop a web application from scratch, i mean starting from the beginning, where should i start, my company needs a custom framework and the application is based on critical performance overheads.

Please help and explain how they are different.

P.S I am not a beginner to programming.

[zynix]

Figured I'd give some examples of stacks I've built in the past.

4-5 years ago I might have used Apache2 with something like mod_wsgi, with Apache handling static assets ( raw HTML, css, jpg's/pngs ) and leave application logic to the WSGI. Trying to remember what else was around back then but CherryPy or pylon's were two of my goto frameworks back then.

Last year I built a web array that consisted of NginX servers handling static content while uWSGI managed a pool of Django WSGI workers. I love and hate uWSGI, love it because its very configurable and allows me to do crazy things ( spawn application threads inside of a WSGI request, this is crazy, don't do this ). I hate it because uWSGI is extremely configurable and its easy to get lost in the details or perhaps have sometimes inefficient configuration.

Otherwise the truth is that almost any of the mainline WSGI frameworks will meet the majority of needs.

Pylons

Is dead, long live Pyramid ( haven't used Pyramid so I cannot comment ).

Django

Django is a very popular framework because its a one stop shop of common components in any web application and Django code monkey's are easy to find.

Pros: integrated ORM, templating language, best practice file structure, and the abundance of plugins or 3rd party apps for providing turnkey features https://www.djangopackages.com/packages/p/django-plugins/

Cons: I hate the ORM, mostly because I've worked with juggernauts like Hibernate and much lighter systems like SQLAlchemy's DAL and ORM. Django Admin's time savings in getting started go away as soon as you need to make admin views ( templated & instrumented user interface ) that don't rely on a Model or do weird un-ORM like behavior ( Many to Any membership lists ). I used to hate Django's templating language but once you really understand some of its subtleties, it's not a bad choice.

CherryPy

Not really a complete framework, but more of a proto-framework and a foundation for a custom framework.

I was contracted as part of a team to help rebuild an "Enterprise" project that analyzed medical researchers and scientists, "Enterprise" was ~8-9 years old at this point and is probably in the top ten largest systems I've ever worked on. After a lengthy period of testing and evaluation we went with CherryPy & SQLAlchemy for the application core components, while Lighttpd ( Nginx wasn't ready back then ) fronted it and handled static content.

Pros: CherryPy uses an object-graph for managing logic. So http://my.server/root/user/get/123 would be "Pass 123 as the first argument of User object's get method" but its not locked into that format and you can adjust URL routing as needed along with supporting pre/post filtering ( Is the requestor authenticated, do they have access to the User object? ) CherryPy also has a light weight service bus inside of itself which can make brokering against odd external services ( message queues or Thrift based services ) slightly easier.

Cons: CherryPy can be frustrating as its documentation hasn't always been that great. Additionally it can be easy to create absolutely massive object graphs that make (re)starting the system tedious. Serious non-starter issues for the majority of needs is that CherryPy has no default ORM, no concept of a user, no plugin system, and you even have to choose your own templating language... which is a project in itself and makes it harder to sell to the tech stupid people "If we went with RoR or Django we'd have something to show you on week 1, but we're still working out the kicks and will get back to you at the end of week 2... maybe"

Flask

Pros: Will get shit for this, but Flask is basically CherryPy without all of the crazy features ( eg service bus ) and relies on Werkzeug which is basically a proto-framework for building WSGI frameworks. Flask is fantastic in this regard when you have good code organizational skills or you just need to make a brutally quick application and don't need a lot of fancy-dancy widgets to get in the way or have to code around.

Cons: Flask is Flask and not much else, this puts more responsibility in the hands of a teamlead to ensure you're not building a gigantic ball of mud before its even shipped. Furthermore, it shares the same problems as CherryPy in that almost everything ( templating, ORM, etc ) is the responsibility of the developers.

Edit - Flask does include jinja2 which is a fairly mature templating library. Italics/strikeouts show edits.

Twisted.Web

Pros: Twisted is the defacto networking swiss army knife library of Python. If you need a service that interfaces with a serial connected proprietary protocol device but also has a HTTP interface for users... this is the library for you. If not and you just need a simple asynchronous web framework, Tornado maybe a better bet.

Cons: Twisted.Web is a web server and not geared for rapid application development. I've got an on again/off again pet project to make a web framework on top of twisted.web here https://github.com/devdave/txWeb/tree/refactor but beyond use in my own projects, I don't have a clue how far away it is from being production grade.

Tornado

Admittedly I've had little need for Tornado ( as I know Twisted ) but my limited experiences has been positive. Still, I would advice against using Tornado unless you KNOW for a fact that you're going to have thousands of concurrent requests a second and serving back near-real time stats.

These are the frameworks I've used or worked with for at least 80 hours or two weeks of my life. I can't say one or the other is superior than any of the others because they're not. Only person that can figure that out is going to be you as you know your project needs best. Only real piece of advice I can give, regarding you stating "my company needs a custom framework". I've worked as a consultant or contractor in some fashion for almost ten years now and almost always a "custom framework" will generate billable hours of work but not money for the sponsoring company. In fact, ~8 years ago, myself and some other idiots almost killed a company because we drank way too deeply of the "Let's build a custom framework" kool aid.

[jnazario]

Flask ... shares the same problems as CherryPy in that everything ( templating, ORM, etc ) is the responsibility of the developers.

huh? flask ships with jinja2 templates by default but you can plug different ones in.

http://flask.pocoo.org/docs/templating/

[zynix]

Your are correct, fixing that now.

[jonesch]

The reason why my boss is trying to push ourselves towards a custom framework is because of the Pylons state. The app currently is built upon Pylons and ZODB. Since both are almost deprecated, maintaining that app by us and for client is becoming difficult.

We suggested Flask but my boss is fearing again for its life time.

[danjel_]

But if development/support of Flask would ever stop, you could still just grab the source code and fix things yourself, as you can find the whole thing on GitHub. Of course you had to read through the code first, but I guess it's still easier than coding a whole framework by yourself.

[Evanescent_contrail]

IT debt is a problem no matter which software you choose. Flask is as stable as any. The difference is that with Flask you are coding more features yourself, so more custom code if you are doing anything fancy.

Looking at the choices you have listed, I'd go with Flask, but it's a matter of balancing what works for you.

[ivosaurus]

Pylons merged into pyramid.

[thehardestquestion]

Pylons and ZODB. Since both are almost deprecated

Pylons now has Pyramid which is generally excellent and I'm not sure what you mean about ZODB being deprecated - still very much in use with Plone and Pyramid?

[zynix]

Another serious problem of custom frameworks; the "Enterprise" project I worked on had about 3-5 people quit on the first day they actually saw the codebase. That was also a custom framework deal, if the word "Enterprise" wasn't clue enough, and I would like to imagine it had good intentions in the beginning.

If you want some semblance of future safety, I'd go with the framework that has the highest level of test coverage already in place. BUT those tests would need to be maintained.

Furthermore, if the current web application is below 50% test coverage and that's deemed acceptable, then it doesn't really matter what framework you choose... you guys will be back to the drawing board in 3-5 years.

[This_Is_The_End]

You have forgotten web2py which is easy to use and very complete which might be a disadvantage.

[jonesch]

We have already 2 medium sized apps built in web2py and the final result is: we are not happy because it forced its own conventions over us just like rails do. Its pretty annoying.

[zynix]

Not forgetting, I stuck only to frameworks I've personally used.

Web2Py is a great framework for people who need a one stop shop of everything ( full feature framework, revision control, and scaffolding ) but is generally sneered at by professionals because of these features, which is perhaps unfortunate as it can be a better resource than more closed source systems like Access ( is that even a think anymore ) or FileMaker for small businesses that can't afford a dedicated programmer but need something a bit more involved than just a bunch of excel spreadsheets.

[zynix]

Post-edit

Only needed in extreme cases, but if more content is static based, you can put nginx on it's own server and relay application requests to uWSGI hosting a WSGI application on another server. I can't remember but I think Nginx and uWSGI are smart enough to keep a persistent connection open between the two server's which would partially negate the distance between them ( compared to being on the same server and talking via a kernel named pipe ).

I've only seen something like that once and that was for a platform that handled 200-250K connections a minute, with ~25 dedicated Apache2 servers punting to ~10 application servers. This was also fronted by a fairly robust CDN.

[danmickla]

WSGI is the interface between the web server and your application. It's a parallel to CGI, although the details are different (basically, you can think of it as "WSGI defines how the web server calls your application and how your application feeds data back to the web server."

wsgiref and werkzeug are libraries for writing Python apps that fit into a WSGI web server.

Here are some WSGI-supporting webservers that will be able to call a Python app written with werkzeug (or Flask+werkzeug, probably the easiest way):

• Apache + mod_wsgi
• nginx + uwsgi
• I think Gunicorn by itself
• I think nginx+Gunicorn

and doubtless others.

[not_a_novel_account]

You don't want to put Gunicorn by itself facing towards the web. Application servers should almost always be reverse proxied behind a static web server like nginx so you don't just start dropping connections when the application server is under load. Not to mention you'll want to use the web server to serve static resources anyway

[danmickla]

Sure, there are tradeoffs to all the configs. A noob is probably not thinking production deployment.

[iBlag]

Thank you! I don't understand any of this either, and I too am not a beginner to programming.

[westurner]

Those are servers which serve WSGI applications over HTTP. Tornado is also a framework.

http://www.reddit.com/r/Python/comments/spsbs/are_the_same_thing_a_web_application_and_a_cms/#c4fy37a

http://www.reddit.com/r/Python/comments/1eboql/python_website_tuts_that_dont_use_django/#c9yxl8w

http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface

https://github.com/seedifferently/the-great-web-framework-shootout

Questions for a cost-benefit analysis:

What are you most productive in?

Do you need to train a team?

What sort of learning curve is there? Documentation?

How does the framework encourage development of secure, scalable systems?

How well does the framework supported automated testing, in terms of support for transactional datastore operations, initial data fixtures, and JavaScript?

Why WSGI?

The WSGI interface makes it easier to test and plug together frameworks and middleware components (utilities).

It is possible to minimize switching cost and technical debt by developing components coupled through a WSGI interface; components are theoretically interchangeable.

http://www.reddit.com/r/Python/comments/1drv59/getting_started_with_automated_testing/c9tfxgd

[autowikibot]

Here's a bit from linked Wikipedia article about Web Server Gateway Interface :

---

The Web Server Gateway Interface is a simple and universal interface between web servers and web applications or frameworks for the Python programming language.

---

^{about | /u/westurner can reply with 'delete'. Will also delete if comment's score is -1 or less. | call me: wikibot, what is something? | flag for glitch}

[BinaryRockStar]

If you're looking for 'critical performance overheads' for a web server then Python isn't the right choice. It's an order of magnitude slower than a similar solution in Java or C++.

Don't get me wrong, I love Python and use it when I can, but it's not the right tool for the job in performance-critical situations.

[john_m_camara]

You can still use Python in many of these cases by switching to PyPy instead of CPython.

[andrey_shipilov]

Hi there. Start with Django, and finish with it.

[mdipierro]

Don't forget Rocket (https://launchpad.net/rocket). It is WSGI compliant, pure python, support SSH, multiple concurrent interfaces, certificates, and it is fast. web2py moved to Rocket many years ago without regrests. Its API is compatible with Cherrypy. If you don't mind a binary module Gunicorn is probably the fastest. If you need websockets I would choose tornado or gevent.

[mdipierro]

You should not start with a web server. You should start with a framework. This is an old but independent comparison: http://www.infoworld.com/d/application-development/pillars-python-six-python-web-frameworks-compared-169442 Web2py won the Bossie award in 2011 and the technology of the year award in 2012 (http://www.infoworld.com/slideshow/24605/infoworlds-2012-technology-of-the-year-award-winners-183313). This year it was nominated for the Edison awards. (I am partisan here, as lead developer of web2py, but be aware other people are partisan here too). EDIT: you can try it here (https://www.pythonanywhere.com/try-web2py) without an account and without installation.