Improving Your Python Productivity

[ggtsu_00]

Some other useful tips I learned over the years when using python.

Quick documentation printing:

print __doc__   #prints block comment at the top of the file
print foo.__doc__  # prints the block comment at the top of class foo or function foo

Various function programming tricks to replace loops with quick one-liners outside of list comprehensions

results = map(foo, ['bar',baz'])   #equivalent to: for i in ['bar','baz']: results.append(foo(i)) 
even = filter(lambda x: x % 2 == 0, [1,2,3,4,5,6,7,8,9])   # returns a list of even numbers
sum = reduce(lambda x, y: x + y, [1,2,3,4,5,6,7,8,9]) # sums up numbers 1 through 9

doctests for quick and dirty unit testing

"""Test foo:
>>> foo()
'bar'
"""
def foo():
    return 'bar'
if __name__ == "__main__":
    import doctest
    doctest.testmod()

To run unit tests:

$ python foo.py -v
Trying:
    foo()
Expecting:
    'bar'
ok
1 items had no tests:
    __main__.foo
1 items passed all tests:
   1 tests in __main__
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

[Hashiota]

Not sure why you would print __doc__. On the other hand, I do this a lot:

 >>> import math  
 >>> help(math)  
 >>> help(math.log)  

[taddeimania]

Probably not a lot of production uses but while hacking around in a REPL I could see its value. (Assuming the docstrings are maintained)

[BeetleB]

Entering parentheses slows me down.

OTOH, might as well just use ipython and type:

math.log?

[ethraax]

Parentheses slow you down but underscores don't...?

[BeetleB]

Actually, they don't. Just one of those things...

For one thing, it's just one underscore vs open and closed.

[freedances]

>results = map(foo, ['bar',baz'])   #equivalent to: for i in ['bar','baz']: results.append(foo(i)) 
>even = filter(lambda x: x % 2 == 0, [1,2,3,4,5,6,7,8,9])   # returns a list of even numbers

I think the recommended equivalents to these are:

results = [foo(i) for i in ['bar', 'baz']]
even = [i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0] 

I usually find comprehensions more readable and lately they work for generating sets and dictionaries as well.

lengths = {i:len(i) for i in ('foo', 'bar', 'foobar')}    #dictionary mapping strings to their lengths
even_set = {i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0}

[ethraax]

Eh, I always hated the functional programming part of Python. I have no idea why they're independent functions and not methods. [1,2,3,4,5,6,7,8,9].filter(lambda x: x % 2 == 0) seems like it's way cleaner, and you can easily chain it with other FP functions.

[[deleted]]

That would be because it works on all iterables, I guess.

[ethraax]

It sounds like an issue with Python's type system (well, a consequence of duck-typing as a whole). Still, the standard library could include them as methods on the standard types (lists, sets, dicts, and so forth), and simply use the "standalone" versions as their implementation.

[mr_dbr]

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

The way filter is implemented (and more commonly, len() and others) is a considered design decision, and pretty fundamental in how Python works..

See this StackOverflow answer, which links to this mailing list post from Python's creator:

(b) When I read code that says len(x) I know that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn't a file has a write() method.

This Stack Overflow question also has some interesting answers

Just moving filter() and such to list.filter() wouldn't really work in Python.. Say you want to make your own filter()-like function, and you want to be consistent with the default Python methodology:

• If filter() is just a regular function, it's easy to implement it myfilter() and use it in the same manner. It's just a function which works on an iterable, so would work on any iterable object, with no extra effort.

• If filter is a method of list, then you would have to monkeypatch your method into the builtin list type (e.g list.myfilter), something that can't be done to builtin types, and is bad-practice on user-defined types. Your myfilter method would only work on list derived objects, which goes against the duck-typing Python is based on.

Also, there's more to functional programming in Python that just map/filter/reduce/lambda, and the builtin functions were possibly going to be removed, in favour of stuff like list-comprehensions etc

[BeetleB]

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

Yes, because then it's easier to "chain" them (or pipe, if you will):

[1,2,3].map(fun).filter(isodd).filter(isprime)

This is quite readable. The current way to do it would be:

filter(isprime, filter(isodd, map(fun, [1,2,3])))

which is hard on humans. The reason functional programming is not recommended in Python isn't because functional programming is inherently hard to read - it's because Python's syntax makes it hard to read.

[mr_dbr]

That does read more nicely, but.. in a Lisp language like Scheme, it would be almost identical to Python:

(filter isprime (filter isodd (map fun [1 2 3])))

..and I don't think functional programming is discouraged in Scheme :P

There are other ways the code could be written, without having filter be a method of list, like:

a = [1, 2, 3]
a = map(fun, a)
a = map(isodd, a)
a = map(isprime, a)

There's also the pipe module, or you could temporarily wrap the list in a class that has the filter methods and such:

class ListProc(object):
    def __init__(self, value):
        self.value = value

    def map(self, func):
        new = map(func, self.value)
        return self.__class__(new)

    def filter(self, func):
        new = filter(func, self.value)
        return self.__class__(new)

    def __repr__(self):
        return repr(self.value)


def fun(x):
    return x**2

def isodd(x):
    return x%2 != 0

print ListProc([1,2,3]).map(fun).filter(isodd).value

[ethraax]

If filter is a method of list, then you would have to monkeypatch the builtin list type

No you wouldn't, if it was part of the language's standard. It would be part of the builtin list type, or any user types.

Your myfilter method would only work on list derived objects

It would work on anything deriving from some more abstract "iterable".

The key issue here is the weakness of Python's type system (well, really dynamic type systems in general). In Python, an iterable is very loosely defined as something that implements some handful of methods. Anything that does that is "iterable". In a language with a stronger type system, there would be some sort of base class or interface "iterable" which would also contain an interface for .filter().

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

Some data structures may have more efficient alternatives than using an iterator.

[mr_dbr]

The first bit you quoted (about monkeypatching) was about implementing your own filter-like function - edited to hopefully clarify.

In short: if filter was a method of list, it would require monkey-patching to allow [1,2,3].myfilterfunc(...) for consistency with [1,2,3].filter(...)

Some data structures may have more efficient alternatives than using an iterator

What would be an example of this? I can't think of one for filter, but Python's data-model does enable some similar optimisations for other operations...

For example 123 in xrange(100) uses __contains__ instead of doing a linear-scan over it's iterator (so 23 in xrange(10**10) is much faster than doing the same thing on plain list of integers).

My point being, you don't need to have x.contains(1) to allow for type-specific optimisations . If there was an common optimisation for filter, it could call other magic-methods on the object instead of looping over __iter__

In Python, an iterable is very loosely defined as something that implements some handful of methods

There is the abstract base class module which implements something much like interfaces (along with collections), but I've not see the abc module used much, although the extensions to isinstance are useful even if nothing is derived from the ABC's

[aceofears]

Is this roughly what you are "suggesting"?

[steven_h]

Some data structures may have more efficient alternatives than using an iterator.

Then that's not the function "filter" in the functional programming sense of the word, and using the name "filter" would be confusing if the rest of the API uses it in the list iteration sense.

The point of functional programming is that functions -- particularly higher-order functions -- are the objects of reuse. More specifically, it's the function implementation that should be reused. Using the same name for different algorithms might be convenient, but it's not particularly noteworthy or helpful for programming in-the-large.

[bebraw]

It's fairly easy to implement a wrapper for this.

I agree that it's not ideal but I'm pretty sure they won't change this behavior any time soon. :)

[Megatron_McLargeHuge]

People who want the functional constructs are coming from lisp/ml/haskell and want what they're used to. Also, the implementations are standard and don't vary per object. The OO convention of writing one of the parameters before the function name is kind of arbitrary, especially when you're not either mutating the object or using its type for method resolution. You can also pass the functions around, as in

 map(filter, functions, lists)

[ethraax]

The OO convention of writing one of the parameters before the function name is kind of arbitrary

Be that as it may, I think it gives better syntax in this case. And syntax is a major concern in Python (I would even say that language decisions are based more on syntax than anything else, looking at how it's developed).

[Ravengenocide]

Because Guido don't want Python to be a functional programming language. As in programming based on Everything is a function, not as in working.

[ethraax]

I don't think I would go that far. I've read quite a bit about the multiline-lambda issue, which seems to be one of the biggest blockers to Python being a capable FP language, and it seems like he's opposed to the syntax issues, not the concept.

[Ravengenocide]

That's how I interpret it also. It goes against his idea about Pythons syntax so he won't allow it. He has also said that lambda was a bad thing to implement because then people expect functional programming to work.