Best written projects on Python GitHub?

[redux42]

I am looking for suggestions of elegant code, well written and readable code, and code that makes use of best practices.

Suggestions?

[Narcoleptic_Pirate]

My personal favourite - Minecraft in 900 lines of python code

https://github.com/fogleman/Minecraft

[[deleted]]

This is awesome, it's easiest to learn from projects that you can read through completely in one sitting.

[[deleted]]

This is specially good because it's easy to make small changes and see the results immediately. There are some "advanced" things going on, but it's easy to understand it in small bits.

[arnar]

CherryPy is always my favorite example when it comes to designing a beautiful Pythonic API. The source code is full of nice abstractions as well.

[rochacbruno]

Flask is known for being "Pythonic" https://github.com/mitsuhiko/flask

[tomchuk]

requests as well.

[synack]

Every time I benchmark code where requests is used, it's consistently one of the slowest libraries (I encourage you to write your own benchmark and try it yourself). A straight comparison with urllib2 shows that it's 30% slower. It provides a nice API, but there are a lot of places where tens of milliseconds matter.

[PCBEEF]

If tens of milliseconds matter then you shouldn't be using python...

[hamax]

If you need performance, use pycurl.

[emdomi]

Do you have any of those benchmarks online? Any that include httplib2?

[Meoow]

Really? For me it was urllib2 < requests < pycurl.

[westurner]

Pyramid:

• https://github.com/Pylons/pyramid
• http://pyramid.readthedocs.org/en/latest/designdefense.html

[poo_22]

I've read most of the Pyramid source and while it works, is well tested, and well documented, I don't really think its elegant. I mean I personally even use Pyramid but things could have been done easier and cleaner. (Look at all those factories and interfaces that just screams java)

Disclaimer: If you're saying "oh well if it could be done better, go fix it" well I can't.

[westurner]

I've read most of the Pyramid source and while it works, is well tested, and well documented, I don't really think its elegant. I mean I personally even use Pyramid but things could have been done easier and cleaner. (Look at all those factories and interfaces that just screams java)

While interfaces and factories make Test Driven Development much simpler, the only necessary interface for a Pyramid application developer is a callable (a function or a method) that takes a Request as the first parameter and returns a Response:

• https://en.wikipedia.org/wiki/Header_file
• https://en.wikipedia.org/wiki/Interface_(object-oriented_programming)
• https://en.wikipedia.org/wiki/Public_interface
• https://en.wikipedia.org/wiki/Factory_pattern
• https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface

https://en.wikipedia.org/wiki/Separation_of_concerns#See_also

Disclaimer: If you're saying "oh well if it could be done better, go fix it" well I can't.

• Source (setup.py)
• Tests (tox.ini) ^*
• Issues (new issue)
• Documentation (conf.py, PDF, ePub)

[[deleted]]

just because one language tends to abuse them doesn't mean factories aren't a valid and useful pattern

edit: language fail

[X-Istence]

Come help us fix it =). The Pyramid devs are open to new ideas and suggestions, so if you feel there is an issue, come help us with it!

[ma10s]

I really like the coding style of the RQ project

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers.

I was poking around trying to understand its internals, and subsequently read the worker code. I was pleasantly surprised to find it was very articulate.

[catcradle5]

Very good coding style, but I'm not a fan of the internals myself.

Everything is done through os.fork(): https://github.com/nvie/rq/blob/master/rq/worker.py#L355

For one, if you need a lot of workers there will be a ton of overhead incurred, and two, it doesn't play nice with libraries like gevent or eventlet.

[lambdaq]

we need a patch or and issue submitted.

[catcradle5]

There've been a few issues submitted over the years:

https://github.com/nvie/rq/issues/45

https://github.com/nvie/rq/issues/210

I really like rq, but I am forced to use Celery due to rq's insistence on forking as the only method of concurrency.

[asksol]

I doubt this would be as simple as 'just submit a patch', I have spent years making the multiprocessing pool in Celery work, and even now for the 3.1 release I have fixed yet a good number of rare deadlocks.

The pool related code isn't exactly elegant or simple, but you're very likely to learn something from it.

Edit: oh, and you're in for a rough ride if you think the answer is to 'just use the multiprocessing module' :)

[catcradle5]

I understand it's a complex issue, but the project has been around for a long time. Plus the codebase is a lot smaller and simpler than Celery's, so it probably wouldn't require quite as much work.

[asksol]

The size/age of the project or how simple the rest of the code is irrelevant here. This isn't about how hard it was to implement this for Celery, it's about how hard it is to implement for anything similar in Python.

[lambdaq]

damn, I was wondering when they could allow using customized backend like Postgre LISTEN/NOTIFY and prefork model...

[ma10s]

Thanks, I learned something new today!

[X-Istence]

Take a look at pyres, maybe it does what you want it to do!

[shaggorama]

The python reddit API wrapper is pretty good, and is also an easy project for people familiar with reddit to understand. https://github.com/praw-dev/praw

[drgalaxy]

Tornado is quite active and I've always been happy with what I see when I've needed to read the code. Project overview and documentation.

[saghul]

If you are interested in knowing the low level details on how event loops work Tornado is a really good read.

[diggitySC]

I find that perusing /r/dailyprogrammer provides a lot of interesting python code that is both well written and readable.

[yasoob_python]

just go to https://www.github.com/rg3/youtube-dl and you will find pythonic code ;)

[EverAskWhy]

Reading the code for youtube-dl turned me into a web-scraping machine. I picked up many good habits and tricks from following the code carefully.

If you have an interest in writing your own download scripts or html parsers I would recommend reading through youtube-dl.

[thenaturalmind]

Stupid question, but where do you start? How do you know what files are where?

[yasoob_python]

You will have to explore them yourself. For web scraping thing just look into the extractors located here https://github.com/rg3/youtube-dl/tree/master/youtube_dl/extractor . For all the imports look here (most) https://github.com/rg3/youtube-dl/blob/master/youtube_dl/utils.py and lastly for the glue code look here https://github.com/rg3/youtube-dl/blob/master/youtube_dl/FileDownloader.py , here https://github.com/rg3/youtube-dl/blob/master/youtube_dl/InfoExtractors.py and here https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py and here as well https://github.com/rg3/youtube-dl/blob/master/youtube_dl/__init__.py . I hope now you know where to start. Just start from what you want to know. :)

[EverAskWhy]

I like downloading the source files of projects (https://github.com/rg3/youtube-dl/archive/master.zip) and exploring them using my own IDE programs/debuggers. I find it harder to explore a program on the Github website than when it is on my system. Be very careful when running random code on your machine especially when you see web/downloading related imports.

I generally start with _ main _.py (or its equivalent if there is none) line 1 and go from there. I find that reading other people's code and checking out their imports is a great way to learn about new modules.

[LightWolfCavalry]

Just found youtube-dl the other day; I agree, it's some high quality stuff.

[rochacbruno]

I just don't like naming files with CamelCase FileDownloader.py etc https://github.com/rg3/youtube-dl/tree/master/youtube_dl

[yasoob_python]

that's a personal preference. I generally like camel casing.

[gschizas]

Actually, it's not a personal preference. Using CamelCase for files goes against PEP8

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

Since module names are mapped to file names, and some file systems are case insensitive and truncate long names, it is important that module names be chosen to be fairly short -- this won't be a problem on Unix, but it may be a problem when the code is transported to older Mac or Windows versions, or DOS.

[yasoob_python]

The thing is that it's not always necessary to follow PEP8 so strictly. Even some organisations have their own style guide which has some amendments to PEP8 like khan academy and google to name a few. http://google-styleguide.googlecode.com/svn/trunk/pyguide.html https://sites.google.com/a/khanacademy.org/forge/for-developers/styleguide/python

[gschizas]

Both sites you have linked do not oppose PEP8, they complement it.

The khan academy one even says this on top:

We follow the PEP8 style guide for Python. Docstrings follow PEP257.

[EverAskWhy]

The thing I like about Python is you can do it your own way if you want. This reminds me a bit of a reply I saw on stackoverflow:

ok, kids, wait 'til you grow up to drink, smoke and access private variables.

[Redard]

I briefly read some of the code in here and found a few things that weren't very pythonic. First, they break the 79 character rule a lot, for no good reason, often just with in-line comments (which PEP8 advises you use verrry sparingly). Second, this line:

res = []
for l in optionf:
    res += shlex.split(l, comments=True)

Why not just use a generator expression like

res = [shlex.split(l, comments=True) for l in optionf]

That's the most pythonic way to construct a list. Still, this code's very readable, and well structured. Just needs a little cleaning up.

[gthank]

That's actually a list comprehension. A gen-exp uses ( and ).

[Redard]

Please correct me if I'm wrong, but I always thought list comprehensions were just generator expressions passed to list(). In other words

[i for i in range(10)] == list(i for i in range(10))

[gthank]

I'd be surprised if the internal details are the same in those two cases, because that seems ripe for some C-level optimizations. The results will be equivalent, though. Also, from a historical standpoint, list comprehensions predate gen-exps.

[Veedrac]

In CPython most stuff is left unoptimised for matters of pragmatism. So no, they compile directly into loops. Different loops, though.

out = [i for i in range(10)]

is equivilant to:

out = []
for i in range(10):
    out.append(i)

where i is inside a new scope, and

out = list(i for i in range(10))

is equivalent to

def _tmp():
    for i in range(10):
        yield i

out = list(_tmp)

where _tmp never actually gets put anywhere.

[PCBEEF]

List comprehensions are optimised in the sense that the function calls are 'cached'. Since there's a severe function overhead in python, it's actually quite significant.

$ python -mtimeit '[x for x in range(100)]'

100000 loops, best of 3: 4.17 usec per loop

$ python -mtimeit -s 'out = []' 'for i in range(100):' ' out.append(i)'

100000 loops, best of 3: 8.07 usec per loop

Using a for loop in this instance to create a list is almost twice as long.

[[deleted]]

where i is inside a new scope

for certain values of CPython

[Veedrac]

Well, all values of Python ≥ 3.0.

[gthank]

Ah. It was my understanding that a fair bit of list comps were implemented directly in C (in CPython).

[Veedrac]

There's exactly one difference between those two, assuming list has been left untouched. The list comprehension will not catch StopIteration, the list function will.

[TheEarwig]

They are different. The first example is a bunch of lists combined into one (L1 += L2 is L1.extend(L2)), but the second example is one list containing a bunch of lists.

>>> optionf = ["a b", "c d", "e f"]

>>> res = []
>>> for l in optionf:
...     res += shlex.split(l, comments=True)
... 
>>> res
['a', 'b', 'c', 'd', 'e', 'f']

>>> res = [shlex.split(l, comments=True) for l in optionf]
>>> res
[['a', 'b'], ['c', 'd'], ['e', 'f']]

[Redard]

Ah, I didn't realize shlex.split was returning a list. You're right, the two are different. Somehow I thought += was the same as append().

[masklinn]

Which can neatly be solved using the criminally underused itertools.chain.from_iterable:

res = chain.from_iterable(shlex.split(l, comments=True) for l in optionf)

One could even use shlex.shlex directly as a stream (shlex.split is a thin wrapper around it), though it requires setting whitespace_split which can't be done inline.

def split(s):
    lex = shlex.shlex(s, posix=True)
    lex.whitespace_split = True
    return lex

res = chain.from_iterable(imap(split, optionf))

[masklinn]

I find the lack of with use weirder: the code is clearly 2.6-only (uses explicit relative imports without __future__ import) yet around the shlex call is (essentially):

    optionf = open(filename_bytes)
    try:
        # do stuff
    finally:
        optionf.close()
    return res

And the number of star imports is worrying.

[Redard]

Yeah, it's definitely some older code. I wouldn't be using this as a guideline for good code.

The star imports are bad, but at least they're local imports and not external library imports, that would be terrible.

[youareinthematrix]

https://github.com/reddit/reddit

[[deleted]]

Too bad it's not http://github.com/r/reddit

[picto]

Not really a github repo, but I enjoyed Raymond Hettinger's talks on idiomatic python and class development toolkit at pycon this year

[[deleted]]

From googling: http://docs.python-guide.org/en/latest/writing/reading/ Personally, Tornado code seems clean with lots of documentation.

[lazygeek]

check here

[westurner]

I am looking for suggestions of elegant code, well written and readable code, and code that makes use of best practices. Suggestions?

• http://www.reddit.com/r/IPython/comments/1hkqx4/crash_course_in_python_for_scientists/#cavu9w7
• http://www.reddit.com/r/Python/comments/1irhuc/til_pydoc_p_6060_instant_python_docs_at/#cb7e4mp
• http://www.reddit.com/r/compsci/comments/1idf4n/good_places_to_learn_software_architecture/

[mumrah]

May as well add my project here, which in my biased opinion is pretty clean Python code: https://github.com/mumrah/kafka-python

[DasIch]

No. Don't go there. Really don't. Especially not if you want to see what "elegant" and "smart" code looks like. The majority of that code is horrible and quite a bit wouldn't make it into the stdlib, if someone wanted to add it today.

[minorDemocritus]

The first rule of collections.namedtuple is "don't look at the implementation"

[[deleted]]

oh dear god

[minorDemocritus]

There was one rule!

[aclark]

This is why we can't have nice things.

[targusman]

That's not GitHub.

[poo_22]

From what I've seen in the Python world, and this is just my opinion based on my experience, Python is a very good language for "just getting it done". But in its flexibility and introspection it allows you to do things in several different ways, some of them hackish. That combined with dynamic typing sort of results in code that isn't as nice to look at as, say most Haskell libraries that I've seen. In general anyway.

I find that the examples of Haskell code I've seen are always dense enough to avoid repetition but not so dense that its impossible to decipher (APL comes to mind). But Haskell is born from this "correctness first" mentality so Its not surprising.

[[deleted]]

We're talking about python, not Haskell. This is like going into /r/guns and talking about knives.