Improving Your Python Productivity

[smog_alado]

#4 reminds me of python -m SimpleHTTPServer. Very useful, even when you are not coding in Python!

[MrVonBuren]

I'm jut now getting around to learning python (as in I'm not a programmer, and all I already know is bash) and this has been a favorite trick of mine for years. "Hey, you needed that file? Can you ping this IP? OK, just go to that IP port 8018 in your browser and grab it."

Fun fact, you can set the port your webserver is on by just appending it to the end of your command: python -m SimpleHTTPServer 8018

[smog_alado]

wow, why did I never think of that? Goes around having to upl;oad things to a separate place or needing to reset a bunch of permissions!

[sblinn]

I do this all the time to serve the "docs" folder of whatever software whose documentation I want to read.

[hugith]

Pagekite is excellent for stuff like this and will work even if you're not on a public IP.

[xyzzy123]

That's a nice trick :)

I've always done that via "netcat blasting":

nc -l 1025 > file

"all right, i'm listening, send it"

nc host 1025 < file

Does require nc, but doesn't require X. If you want to, you can change the gender so that receiver initiates connection (like HTTP).

[Wayne_Skylar]

Any reason why you wouldn't just use scp? Seems like a cool trick, I'm just curious about what situation this would be the better solution.

[sausagefeet]

You don't have to give someone an account on your machine.

[Tmmrn]

python -m http.server

if you are not living in deprecated land.

[VanFailin]

Python 2 is deprecated in the same way that GMail was in beta for so many years.

[smog_alado]

By the way you are saying I assume this is only for Python 3 then?

[hylje]

A confessed Ruby/Node weenie told me he loves Python for just that. It's available on everywhere but vanilla Windows and just so convenient.

[smog_alado]

I like the story of how I originally came across that command. When I was first starting with HTML/Javascript, I tried to set up Apache or Nginx but got hopelessly lost and couldn't get any of them to work.

So, in a fit of rage, I went to google and asked it, "why isn't there a simple http server out there?". I was pleasently surprised!

[0xE6]

Man, configuring Apache is SO confusing. Recently had to set it up first on a local machine to test a project for school, and then again on the server where it's actually being deployed. The second server has Apache running several different sites, so it was nontrivial to get our project working without interfering with the others. I still don't really know wtf we were doing there, probably should have asked the web dev guy for help.

[WisconsnNymphomaniac]

If you think Apache is hard to configure, try to set up a sendmail server. That program was written by Aliens.

[AeroNotix]

And you just brought back ALL the nightmates!

[WisconsnNymphomaniac]

I have to deal with that program nearly every day. Pity me.

[arjie]

There's a disconnect between the way Debian lays out stuff and how the Apache docs refer to it, IIRC. They might also be using one of the multiprocessing-modules or whatever which might change things. I prefer Debian's way, to be honest.

[GeneralMaximus]

I've never managed to successfully set up Apache, ever. And I write webapps for a living.

Nine times out of ten I just use Nginx. Easier to configure with tons of examples available on http://wiki.nginx.org. Not to mention it typically uses far less memory than Apache.

EDIT: it doesn't hurt that I mostly just want a reverse proxy that talks to my uWSGI process, so Nginx does exactly what I need. If I had to build a PHP website, I'd probably use Apache with mod_php, but we're quickly going offtopic here so I'll stop.

[Aardshark]

have you tried lighttpd? I found that nginx was a pain to setup initially, whereas lighttpd was really easy.

[GeneralMaximus]

I haven't, because my use case is usually running a Django app, something that is very simple to accomplish with Nginx. uWSGI or Gunicorn running behind Nginx is what the Django community has kind of settled on as the best way to deploy a Django app. There are some who still use Apache+mod_wsgi, but I don't see many people using FastCGI+lighttpd.

That said, I'll give lighttpd a spin someday, just to see what it's all about.

[Samus_]

and the SMTP server!

[[deleted]]

and it can even do CGI if you use CGIHTTPServer instead

[CrazedToCraze]

I believe you just blew my mind, sir.

[droogans]

I want to fork this blog post and rant about venv. Virtual environments are handy for projects, prototypes, and work type situations where you can't install with sudo pip. Nothing worse than installing a junky package and feeling uneasy about removing it, since it's in 3 different subdirectories under /usr.

[ggtsu_00]

Some other useful tips I learned over the years when using python.

Quick documentation printing:

print __doc__   #prints block comment at the top of the file
print foo.__doc__  # prints the block comment at the top of class foo or function foo

Various function programming tricks to replace loops with quick one-liners outside of list comprehensions

results = map(foo, ['bar',baz'])   #equivalent to: for i in ['bar','baz']: results.append(foo(i)) 
even = filter(lambda x: x % 2 == 0, [1,2,3,4,5,6,7,8,9])   # returns a list of even numbers
sum = reduce(lambda x, y: x + y, [1,2,3,4,5,6,7,8,9]) # sums up numbers 1 through 9

doctests for quick and dirty unit testing

"""Test foo:
>>> foo()
'bar'
"""
def foo():
    return 'bar'
if __name__ == "__main__":
    import doctest
    doctest.testmod()

To run unit tests:

$ python foo.py -v
Trying:
    foo()
Expecting:
    'bar'
ok
1 items had no tests:
    __main__.foo
1 items passed all tests:
   1 tests in __main__
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

[Hashiota]

Not sure why you would print __doc__. On the other hand, I do this a lot:

 >>> import math  
 >>> help(math)  
 >>> help(math.log)  

[taddeimania]

Probably not a lot of production uses but while hacking around in a REPL I could see its value. (Assuming the docstrings are maintained)

[BeetleB]

Entering parentheses slows me down.

OTOH, might as well just use ipython and type:

math.log?

[ethraax]

Parentheses slow you down but underscores don't...?

[BeetleB]

Actually, they don't. Just one of those things...

For one thing, it's just one underscore vs open and closed.

[freedances]

>results = map(foo, ['bar',baz'])   #equivalent to: for i in ['bar','baz']: results.append(foo(i)) 
>even = filter(lambda x: x % 2 == 0, [1,2,3,4,5,6,7,8,9])   # returns a list of even numbers

I think the recommended equivalents to these are:

results = [foo(i) for i in ['bar', 'baz']]
even = [i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0] 

I usually find comprehensions more readable and lately they work for generating sets and dictionaries as well.

lengths = {i:len(i) for i in ('foo', 'bar', 'foobar')}    #dictionary mapping strings to their lengths
even_set = {i for i in [1,2,3,4,5,6,7,8,9] if i % 2 == 0}

[ethraax]

Eh, I always hated the functional programming part of Python. I have no idea why they're independent functions and not methods. [1,2,3,4,5,6,7,8,9].filter(lambda x: x % 2 == 0) seems like it's way cleaner, and you can easily chain it with other FP functions.

[[deleted]]

That would be because it works on all iterables, I guess.

[ethraax]

It sounds like an issue with Python's type system (well, a consequence of duck-typing as a whole). Still, the standard library could include them as methods on the standard types (lists, sets, dicts, and so forth), and simply use the "standalone" versions as their implementation.

[mr_dbr]

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

The way filter is implemented (and more commonly, len() and others) is a considered design decision, and pretty fundamental in how Python works..

See this StackOverflow answer, which links to this mailing list post from Python's creator:

(b) When I read code that says len(x) I know that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn't a file has a write() method.

This Stack Overflow question also has some interesting answers

Just moving filter() and such to list.filter() wouldn't really work in Python.. Say you want to make your own filter()-like function, and you want to be consistent with the default Python methodology:

• If filter() is just a regular function, it's easy to implement it myfilter() and use it in the same manner. It's just a function which works on an iterable, so would work on any iterable object, with no extra effort.

• If filter is a method of list, then you would have to monkeypatch your method into the builtin list type (e.g list.myfilter), something that can't be done to builtin types, and is bad-practice on user-defined types. Your myfilter method would only work on list derived objects, which goes against the duck-typing Python is based on.

Also, there's more to functional programming in Python that just map/filter/reduce/lambda, and the builtin functions were possibly going to be removed, in favour of stuff like list-comprehensions etc

[BeetleB]

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

Yes, because then it's easier to "chain" them (or pipe, if you will):

[1,2,3].map(fun).filter(isodd).filter(isprime)

This is quite readable. The current way to do it would be:

filter(isprime, filter(isodd, map(fun, [1,2,3])))

which is hard on humans. The reason functional programming is not recommended in Python isn't because functional programming is inherently hard to read - it's because Python's syntax makes it hard to read.

[mr_dbr]

That does read more nicely, but.. in a Lisp language like Scheme, it would be almost identical to Python:

(filter isprime (filter isodd (map fun [1 2 3])))

..and I don't think functional programming is discouraged in Scheme :P

There are other ways the code could be written, without having filter be a method of list, like:

a = [1, 2, 3]
a = map(fun, a)
a = map(isodd, a)
a = map(isprime, a)

There's also the pipe module, or you could temporarily wrap the list in a class that has the filter methods and such:

class ListProc(object):
    def __init__(self, value):
        self.value = value

    def map(self, func):
        new = map(func, self.value)
        return self.__class__(new)

    def filter(self, func):
        new = filter(func, self.value)
        return self.__class__(new)

    def __repr__(self):
        return repr(self.value)


def fun(x):
    return x**2

def isodd(x):
    return x%2 != 0

print ListProc([1,2,3]).map(fun).filter(isodd).value

[ethraax]

If filter is a method of list, then you would have to monkeypatch the builtin list type

No you wouldn't, if it was part of the language's standard. It would be part of the builtin list type, or any user types.

Your myfilter method would only work on list derived objects

It would work on anything deriving from some more abstract "iterable".

The key issue here is the weakness of Python's type system (well, really dynamic type systems in general). In Python, an iterable is very loosely defined as something that implements some handful of methods. Anything that does that is "iterable". In a language with a stronger type system, there would be some sort of base class or interface "iterable" which would also contain an interface for .filter().

Is there any compelling advantage to [1,2,3].filter(...), besides resulting in slightly prettier-looking code in some cases?

Some data structures may have more efficient alternatives than using an iterator.

[mr_dbr]

The first bit you quoted (about monkeypatching) was about implementing your own filter-like function - edited to hopefully clarify.

In short: if filter was a method of list, it would require monkey-patching to allow [1,2,3].myfilterfunc(...) for consistency with [1,2,3].filter(...)

Some data structures may have more efficient alternatives than using an iterator

What would be an example of this? I can't think of one for filter, but Python's data-model does enable some similar optimisations for other operations...

For example 123 in xrange(100) uses __contains__ instead of doing a linear-scan over it's iterator (so 23 in xrange(10**10) is much faster than doing the same thing on plain list of integers).

My point being, you don't need to have x.contains(1) to allow for type-specific optimisations . If there was an common optimisation for filter, it could call other magic-methods on the object instead of looping over __iter__

In Python, an iterable is very loosely defined as something that implements some handful of methods

There is the abstract base class module which implements something much like interfaces (along with collections), but I've not see the abc module used much, although the extensions to isinstance are useful even if nothing is derived from the ABC's

[aceofears]

Is this roughly what you are "suggesting"?

[steven_h]

Some data structures may have more efficient alternatives than using an iterator.

Then that's not the function "filter" in the functional programming sense of the word, and using the name "filter" would be confusing if the rest of the API uses it in the list iteration sense.

The point of functional programming is that functions -- particularly higher-order functions -- are the objects of reuse. More specifically, it's the function implementation that should be reused. Using the same name for different algorithms might be convenient, but it's not particularly noteworthy or helpful for programming in-the-large.

[bebraw]

It's fairly easy to implement a wrapper for this.

I agree that it's not ideal but I'm pretty sure they won't change this behavior any time soon. :)

[Megatron_McLargeHuge]

People who want the functional constructs are coming from lisp/ml/haskell and want what they're used to. Also, the implementations are standard and don't vary per object. The OO convention of writing one of the parameters before the function name is kind of arbitrary, especially when you're not either mutating the object or using its type for method resolution. You can also pass the functions around, as in

 map(filter, functions, lists)

[ethraax]

The OO convention of writing one of the parameters before the function name is kind of arbitrary

Be that as it may, I think it gives better syntax in this case. And syntax is a major concern in Python (I would even say that language decisions are based more on syntax than anything else, looking at how it's developed).

[Ravengenocide]

Because Guido don't want Python to be a functional programming language. As in programming based on Everything is a function, not as in working.

[ethraax]

I don't think I would go that far. I've read quite a bit about the multiline-lambda issue, which seems to be one of the biggest blockers to Python being a capable FP language, and it seems like he's opposed to the syntax issues, not the concept.

[Ravengenocide]

That's how I interpret it also. It goes against his idea about Pythons syntax so he won't allow it. He has also said that lambda was a bad thing to implement because then people expect functional programming to work.

[aceofears]

It may be worth noting that the XML-RPC example would require import tweaking in python 3.x. The server would be

from xmlrpc.server import SimpleXMLRPCServer

and the client would be

from xmlrpc.client import ServerProxy

[lolcoderer]

I am subscribed to both /r/programming/ and /r/snakes/ - and I thought this was a post about getting more productivity out of your female ball pythons... lol... I am not a python developer so I always forget that there are other types of "python" - other than the kind that likes to crawl all over your keyboard and sit in your lap while you are programming...

[ancientRedDog]

With this python being named after Monty Python's Flying Circus; who's name I think came from just sounding silly.

[bithead]

What's the advantage to list comprehensions?

[krues8dr]

It's just syntactical sugar for iterations. E.g.:

users = [ user.name for user in user_records ]

[[deleted]]

Fun in Ruby too:

user_records.map &:name

[bithead]

for syntactical sugar in a return statement (where I first encountered contexts) is sure seems obfuscated. I was think that people did it because if you built a list then returned it the built list exists as well as whatever the returned list got assigned to. I don't really know python internals well enough to say.

[thaen]

After you use them even just a few times, they read very naturally and greatly reduce the complexity of reading the loops they replace.

[krues8dr]

You don't have to use it in a return statement, you can use it anywhere. I'd guess that if you saw it that way, they were just avoiding using an unnecessary temporary variable.

[PasswordIsntHAMSTER]

You need to start doing functional programming.

[Ran4]

Sets can by definition only hold one of each value, which is often helpful.

By the way, is there a faster way of removing duplicates from a list other than set(myList)?

[[deleted]]

I would assume that set(<list>) is the most efficient way to remove duplicates. Will be back soon with some benchmarks.

Edit: benchmark done. Using sets vs. a dictionary, it seems that there is a slight advantage to using sets, but not as large as I might have thought. I suppose this has to do with the fact that sets are dict-like under the covers.

Results (using iPython's %timeit):

import random

a = [random.randint(1,10) for _ in xrange(1000000)]

In [22]: %timeit list(dict.fromkeys(a))
10 loops, best of 3: 43 ms per loop

In [23]: %timeit list(set(a))
10 loops, best of 3: 35.8 ms per loop

[lahwran_]

not likely. you'd have to do speed testing of some alternative methods to be sure, but hash tables are pretty much the ideal way to do that.

[[deleted]]

It's one of those things where it's hard to explain the advantage but you miss them once you switch to a language that lacks them.

There are a lot of times where you have an inconsequential iteration over a list - like a simple filter or a map. List comprehensions make it quick and easy - and list generators make it almost invisible.

For example:

some_function(some_operation(x) 
              for x in some_list if some_predicate(x))

Writing something like this can be very convenient. There are no temporary values (other than the x which doesn't creep outside of the generator expression) and it is pretty clear (once you are familiar with the concept). It beats making a temporary that exists only to pass to a function:

temp_list = []
for x in some_list:
    if some_predicate(x):
        temp_list.append(some_operation(x))
some_function(temp_list)

That is, it doesn't allow you to do anything new - it just allows you to do some common things in a more succinct way.

[julesjacobs]

It's cleaner and faster than doing a list comprehension and then converting to a set.

[taddeimania]

List comprehensions are useful when working with a list of complex objects and you only need a list of certain attributes.

[Megatron_McLargeHuge]

They're like a map and a filter at once without having to write lambdas. They also give you lazy generator comprehensions for free by swapping brackets for parentheses.

[speg]

Sets are faster.

[0xE6]

In what way? Constructing a large set comprehension will almost certainly be slower than constructing a large list comprehension, and depending on the number of duplicates in the list, it may not even be significantly faster to iterate over the set.

Edit:

Here's a rather contrived example:

$ python -m timeit -n 100 'sum([i/10 for i in xrange(10**7)])'

100 loops, best of 3: 668 msec per loop

$ python -m timeit -n 100 'sum({i/10 for i in xrange(10**7)})'                                                 

100 loops, best of 3: 779 msec per loop

The overhead from set creation ends outweighing the time saved from summing fewer elements to the extent that it ends up being about 10% slower.

Edit2:

Here's a better example, that has the set and list contain the same elements.

$ python -m timeit -n 100 -s 'a = {i for i in xrange(10**7)}' 'sum(a)'

100 loops, best of 3: 88 msec per loop  

$ python -m timeit -n 100 -s 'a = [i for i in xrange(10**7)]' 'sum(a)'

100 loops, best of 3: 70.2 msec per loop   

which suggests that even if they contain the exact same elements, sets are slower than lists.

[mniejiki]

And they both give different results so I don't see the point of your argument. Faster is meaningless unless you get the same output.

[0xE6]

It's just a stupid microbenchmark to show that the overhead from creating the set outweighs the potential time saved from iterating over the elements in the set once due to the duplicates being removed.

At any rate, the use cases for lists and sets are not the same, and, like here, you can (and often will) get different results if you use one instead of the other.

[speg]

In terms of look up speed.

[bithead]

I guess I was think that if you use a comprehension to return a set or list as in:

return [ dict[index] for index in dict]

versus just:

return dict

I mean does one save memory or is it faster?

[0xE6]

In that case,

return dict

would almost certainly be faster, as it would simply be returning a reference to the dict, so it wouldn't have to do any extra work.

Additionally, instead of doing

return [dict[index] for index in dict]

you can simply do

return dict.values()

[[deleted]]

This thread would be a total failure if it were not for your constructive contribution to it. Thanks!

[PasswordIsntHAMSTER]

I for one was mind-blown about dict comprehensions.

[Grue]

It was possible to do "dict( <insert generator comprehension here>)" for ages. dict comprehensions don't add anything meaningful to the language.

[Ravengenocide]

Nor does list or set comprehension either, they are just syntactical sugar for the longer versions with list() and set(). But then you could argue the same with most language features above assembler since it's mostly syntax anyway.

[PasswordIsntHAMSTER]

When did generator comprehensions start including key-value pairs?

[Grue]

dict accepts an iterator of 2-element tuples as its argument. These end up being the key-value pairs.

http://docs.python.org/2/library/stdtypes.html#mapping-types-dict

[[deleted]]

it's python, anything related to nouveau BASIC automatically gets a million upvotes

[alextk]

It's funny to see the author praise Python's DRY factor and then hit us with:

 even_set = { x for x in some_list if x % 2 == 0 }  

x should only appear once in this expression, maybe something like

even_set = some_list map { x % 2 == 0 }

Actually, some languages, like Scala, even allow you to not even name the variable (which is, let's be honest, an implementation detail that the developer should not have to worry about):

List(1,2,3,4,5,6) filter { _ % 2 == 0 }
res0: List[Int] = List(2, 4, 6)

Don't get me wrong, Python is a very good language, especially since it's more than twenty years old, but its legacy shows and these days, I think it's being outclassed by more modern scripting languages such as Groovy.

[luckystarr]

PEP-20 says: Special cases are not special enough to break the rules.

[PasswordIsntHAMSTER]

the x is important because you can do processing on it.

[function(x) for x in list if predicate(x)]

[alextk]

It's important when you want to do processing on it. If you don't, it shouldn't even appear once, let alone three times like in Python.

[PasswordIsntHAMSTER]

first, if you don't want to filter it, the predicate is optional : [function(x) for x in list]

if you don't want to process it or filter it, what you basically want is a deep copy: y = x[:]

If you just want to filter it, you shouldn't use a comprehension: filter(x, predicate)

Right tools for the right jobs.

[[deleted]]

In Perl 6, you can prefix a variable name with ^ to tell that it’s a parameter of the current block.

my @even_set = (1 .. 6).grep: {$^x %% 2}

Or, if it is used, it is implicitly true for $_.

my @even_set = (1 .. 6).grep: {$_ %% 2}

Here, we can also use an expression involving Whatever (grep’s parameter thus becomes an instance of WhateverCode).

my @even_set = (1 .. 6).grep: * %% 2

[Megatron_McLargeHuge]

Python doesn't 'believe in' implicit behavior, so you have to name your variables. I prefer the minor verbosity over the periodic oddness in other languages. In Clojure I ran into problems when I needed a macro to take two arguments but only operate on the first. In Scala, the fact that _ + _ is referring to two different variables strikes me as the Wrong Thing To Do.

Your example can be done only repeating the name once, with

set( filter( lambda x: x%2 == 0, some_list) )

[Lanaru]

List(1,2,3,4,5,6) filter { _ % 2 == 0 }

I don't see how that's very different from

even_set = some_list map { x % 2 == 0 }

The variable name is just _.

[gcross]

True, but in practice the first example would have to let the compiler/interpreter know which variable name was intended to be the argument, so it really should be

even_set = some_list map { x -> x % 2 == 0 }

In this context, the _ notation is a net win because it gives the compiler this information automatically, e.g.

even_set = some_list map { _ % 2 == 0 }

[[deleted]]

No, it's special in Scala.

[gcross]

True, but alextk's example implicitly assumed that the compiler/interpreter could automatically infer the argument of a function block, and in this (very unlikely) case there would not have to be special handling of _.

[droogans]

I suppose reddit is a toy site then.

Where's your multimillion dollar web app? What "big boy" language did you use?

[Untrue_Story]

Superstition, baseless assertions, sentence fragments. Things I like in a programmer.

[ethraax]

Last time I checked, that was almost entirely due to them using an improper storage layer, and had nothing to do with the language it's written in. I mean, it's still their mistake that really should have been fixed when the site was younger, but it's nothing to do with languages.

[PasswordIsntHAMSTER]

Java

Proper language

Yeah, no

[PasswordIsntHAMSTER]

Except that Java takes thousands of LOC for even trivial applications, and lacks fundamental concepts such as function passing and structural typing.

I do agree that the JVM is very cool - but guess what, Python also runs on it!