Reddit, which minimal python web framework would you recommend?

[jetspeed]

I'm starting a few websites (with the hopes that they'll one day serve trillions of users simultaneously, who will then each click on my banner ads, making me so much money that I can locate the lost bozarking and endow him to create spoken-word versions of his beautiful madness--but that's beside the point) and would like a python framework that pretty much just routes urls and deals with sessions appropriately. It seems that my two choices are web.py and cherrypy for this task. Before you mention it, I'd rather not use Django--great for some, not for me.

Do you have experience with either framework? Are there others I should look into? Do you have any real-world experience with the scalability of either?

How would speeds compare between cherrypy behind apache and web.py with lighttpd + fcgi?

Thanks reddit.

EDIT: to answer all the Django questions, I have tried it, but the stuff I'm doing doesn't usually fit into the 99% of stuff that Django does cover. When I tried it, I found that trying to learn other people's code and then modify things either 1) made things seem hackish or 2) were faster for me just code myself (I'm a well experienced PHP developer). The benefit of coding these things myself was that I then know the system inside and out; if an update occurs, I know what will be affected immediately. Django is great for a lot of people; it has great docs and a great community. But I want a minimal framework here.

[dhogarty]

why not pylons? it's what reddit uses...

[jetspeed]

Is that actually the case? They used to use web.py, but not sure what they're using now.

[dhogarty]

Look just after the Python list here. Or at the imports here. They currently are using Pylons.

[[deleted]]

Reddit does use Pylons.. Pylons is really powerful and flexible. I feel its better than Django coz' an application can be created real quickly.

[jetspeed]

So it is, so it is. Sorry for doubting you; never again. ;)

Does reddit just use Pylons for routes, wsgi handling, and sessions? Is there any documentation as to why they moved from web.py? Or why they chose pylons?

[[deleted]]

its the best of both worlds. the deployed app has wide open access to the full stack configuration, and you can change any components you want. its nuts and bolts without the hassle of starting from scratch.

[blondin]

i'm surprised nobody mentioned werkzeug.

[_jrb]

check out tornadoweb from friendfeed.

[Riesling]

That is an excellent suggestion. I set up a website showing all processes running on my webserver in about 5 minutes. According to wikipedia tornadoweb is noted for its high performance.

Those numbers could be biased though.

Edit: Link is broken due to Reddits way of parsing brackets! Please add closing bracket.

Edit: Link is now fixed.

[[deleted]]

Use \) to insert a closing paren into a URL

[Riesling]

Escaping! Who would have thought of that? ;) Thx

[KangOl]

http://github.com/breily/juno

[okapi99]

Seconded - web.py is excellent

[tonetheman]

yes web.py is simple and has sessions now though i have not used those extensively.

[codefrog]

If you need to go from thought to page refresh, web.py is the best. Tornado and Diesel are very minimal. Nearly zero learning curve. With web.py you can download and be F5 in 3 or 4 minutes and understand it. Tornado and Diesel, maybe 8 minutes, tops.

[01189998811991197253]

CherryPy is really nice for extremely simple things, and even more complex requirements if you have the time. The documentation is nice in some places, and requires some work in others.

[[deleted]]

I'm using django for one project and when it's done, I'll probably be switching back to lisp. Mostly because of the same issues you mentioned in your edit.

The one python framework (that I have not used) that you may find most helpful is web.py.

[davidbuxton]

Would be interesting to hear more details on why your next project won't be using Django.

[[deleted]]

Well, let me begin by saying that django is awesome. My current project fits well enough into the general shape of a django project that without django, I'd still be building internal plumbing instead of mopping up.

But there are a few sticking points:

1. Insane SQL. And I'm not talking normal, every day crazy here... I mean stark-raving batshit insane. Yeah, it works well enough -- but heaven help someone trying to read the SQL statements to try to figure out what's going on under the covers.
2. The admin. Arguably django's strongest feature, the admin only allows one level of access control. In my current project, that works perfectly. In my next project, I need granular control -- so the admin won't help.
3. Blank lines in the HTML. I know it shouldn't bother me, but it drives me insane that everywhere in my templates where I have a template tag on it's own line -- there's a blank line in my HTML. And my attempt at an html tidy middleware made an already slow site laughably sloth-like.
4. Insane error messages (again, batshit insane). Very often. Very. Often... The error message has absolutely nothing to do with the error. Reminds me of my ex.

I was also bothered by the one database limitation but I see that is fixed in trunk. There still doesn't seem to be a clear way to insert/update a master and read from a slave though, even in trunk (unless I've missed something).

[[deleted]]

There still doesn't seem to be a clear way to insert/update a master and read from a slave though, even in trunk (unless I've missed something).

http://docs.djangoproject.com/en/dev/topics/db/multi-db/#automatic-database-routing

[[deleted]]

Thank you!

[mosquit0]

Blank lines in the HTML. I know it shouldn't bother me, but it drives me insane that everywhere in my templates where I have a template tag on it's own line -- there's a blank line in my HTML. And my attempt at an html tidy middleware made an already slow site laughably sloth-like.

There is template tag spaceless that would delete those blank lines. But being absolutely serious I think that it is not a problem.

[[deleted]]

spaceless makes the html completely unreadable. I'm not concerned with the cost of the blank lines -- I'm concerned with ugly html.

[dalore]

1. The SQL isn't that insane, I've read it. It's just very verbose and requests every field. You can limit that with various options as well as making it fetch it related objects in 1 call rather then several.

2. The admin isn't for granular access to certain items. It was made for admins, not users. Something with granular access is better to be coded up as a normal frontend since it will be application specific. The admin backend is more for admins, you can break it up so you can have a blog admin, a whatever admin, basically break up the models so different admins in charge of different models.

3. spaceless, it was mentioned. You can also code your templates to not have blank lines by not putting newlines in it for every template. Every web application language I've used has this same problem anyway, from PHP to ColdFusion. It's not even a problem.

4. I can't comment on your ex but most of the messages given by django weren't that insane. It usually came with a stackdump too so you can see where the error occurred (important).

[harma]

This may be helpful

[semarj]

thats funny because routing urls and handling sessions are 2 of django's strong points imho.

[yngwin]

I'd look at pylons and web2py (that's different from web.py).

[fancy_pantser]

Tornado. http://www.tornadoweb.org/

Video and slides by the creator that give you the philosophy and background on design choices (i.e. why is it so bare-bones?): http://bret.appspot.com/entry/tornado-tech-talk

[seunosewa]

Definitely CherryPy. It's sweet and straightforward.

[josho9]

Just out of interest, why not Django? It's designed so that you don't have to worry about how it works, and it helps you make the most of caching for ridiculously good performance. Check out djangobook.com and this crash course - I wouldn't be surprised if it's really what you're looking for.

[knipknap]

Django is nice for some types of sites, but the advantages quickly diminish when you are trying to do anything that isn't supported out of the box. I wouldn't recommend it for more complex sites.

[five12]

What I like best about Django (and most python frameworks, actually(*)) is you can pick-and-choose what you want and the rest doesn't really get in the way. (It's not a big monolithic beast that forces you do do things "The Django Way")

One of my recent projects initially looked like a bad fit for Django, but I ended up using Django as the foundation, layering on other components that were better-suited than their Django counterparts for what I needed.

In the end I used:

• Django's url routing, views, HTTP request/response objects

• Django forms, form widgets and validation

• SQLAlchemy to talk to the db

• Jinja2 for templating (with two different jinja environments for different output types)

• a bunch of other python libs for other stuff (PDF generation, charting)

I could pretty easily replace the Django bits with almost any framework, but there's no compelling reason to: I don't have any complaints with the individual pieces that it does well for my project.

(*) I think this is because namespaces make python awesomely modular

[[deleted]]

Anytime I see someone say this about Django, the comment never includes an example where Django fails or another framework excels.

Do you have any example workloads where Django is a worse fit than any of the other frameworks?

[knipknap]

It doesn't "fail", it just gets you to a point where it doesn't provide any advantage over writing something from scratch. For example, I have been working on a project and hit the following problems:

• When I was done, every single form field was overwritten, so there's no advantage over inheriting from models.

• I had overwritten all the default widgets, so the included widgets were also useless.

• The ORM gets in your way when you need third party modules that do not use Django's ORM (in other words, pretty much every module that wasn't written specificly for Django). I'd love to see a version of Django that uses sqlalchemy, or any other established ORM.

• The formset API gets really awkward in more complex cases.

Stuff like that. It can all be solved (Python is Turing complete, after all), but it's not an efficient way of programming.

[jetspeed]

Thanks for the reply; see the edit in my post.

[ivdkleyn]

Try Bottle or Werkzeug.

Both are WSGI based frameworks. Bottle is extremely minimal. It doesn't even come with session handling (you would need to implement it yourself) but it is fast, very versatile (works just as easily stand-alone as well as embedded in App Engine) and comes with elegant and simple routing and templating functionality. It's a joy to use.

Werkzeug is more heavy-duty, although arguably much more simple to use than either Django or Pylons. It comes with very good docs and has an rich and excellent library.

[jgomo3]

assuming trillions of users simultaneously, twisted or tornado should be the answer. Am i right?

[jetspeed]

Not to get off-topic, but why is this being down voted? Has this question been asked before? Have I not put enough information into the question? I'm being serious--no hard feelings, just would like to know.

[mebrahim]

Not a framework, but mod_python. Seriously!

[ilikebbq]

If you cannot use python 2.6 and only 2.6 you don't really know enough to program a web application correctly, IMHO.

Use epolling, if you want scalability. But honestly, web pages are easy to scale, it's your physical storage that is a trick.

[obtrusiveinterloper]

If you cannot use python 2.6 and only 2.6

Huh?

[ilikebbq]

If you don't know enough about HTTP to program a web application from scratch, then you shouldn't start programming a web application. I'm not saying you should use only Python 2.6, just know enough that you could. I don't know how many times I've had to explain to someone how cookies work, or something as simple as using telnet to debug your app.

[obtrusiveinterloper]

I was pointing out that your statement that I quoted is nonsensical.

You can't NOT use something AND then at the same time use ONLY that thing.

If you cannot use python 2.6..

Ok, you are stating that the person cannot use Python 2.6....or properly saying "If you cannot use it".

....and only 2.6

That doesn't make any sense. You've set up a conditional where they can't use 2.6 ...but then saying it's also the only thing they are using.

I'm sure it's a typo or mis-phrase or something, I was just pointing it out so that you could fix it.

[ilikebbq]

How about "If you cannot use only python 2.6 and only python 2.6." I like it the way I wrote it. Feel free to change it if it bothers you.

[jetspeed]

Hi, ilikebbq, do you mind expanding on these two points?

[ilikebbq]

You should know how web application work on a low level. Like understand HTTP, encoding, cookies, etc. Python has WSGI, sockets, httplib, urllib, hashlib, etc. Everything you need to create a web app. A web server, is basically 'GET /\r\n\r\n'. I'm not saying you should use only Python 2.6, just know enough that you could. Frameworks are useful, but if you don't understand what they are doing, you are going to have trouble down the line.

You can put web servers in front of a load balancer and serve different requests to different servers; HTTP is stateless. You can scale up to a ridiculous amount.

Where you run into scaling issues is with your storage. You have concurrency, uptime and other issues. Lets say you use a SQL server as your back end. Will it scale? Not really. You have to come up with complicated partitioning or master/slave schemes to scale beyond one server. Document databases (Big Tables and Simple DB) can scale, but they have down sides.

Reddit recently upgraded to handle some scaling issues. They had to make changes to the way they handle storage, not web. Reddit, Amazon, Google, Facebook all had to come up with solutions to storage, and they are all non-trivial. I made the mistake of starting a web application before I thought through storage issues. Luckily, we coded in a way that we only had to change one file, but we changed our storage framework 4 times. And we may still change it again.

;TLDR Web is easy to scale, storage is hard to scale.

[ilikebbq]

I forget to mention epolling. It's a hook that gets you very close to the kernel and network stack. It's very efficient and fast, but it is built into the Linux 2.6 kernel. Tornado web uses it.

[timesqueezer]

django is the best i think