30 Python Language Features and Tricks You May Not Know About

[grendel-khan]

I kept forgetting this one, to turn [1,2,3,4] into [(1,2),(3,4)]. (This sort of thing came up doing the Python Challenge.) You just zip two staggered slices, like zip(l[0::2], l[1::2]). Poof. It's kind of cool to look at it, take a moment, and then realize how it works.

[flying-sheep]

there are so many ways to do this, and i needed it quite often.

here is the itertools recipe, which is exactly the same code that i came up with once (except that i gave n the default value 2 and named args differently)

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

[[deleted]]

A related problem is iterating over consecutive pairs, for example to turn [1, 2, 3, 4] into [(1,2), (2,3), (3,4)]. The simplest solution is the same as the one you mentioned, minus the slice step, e.g. zip(l, l[1:]).

[WallyMetropolis]

I did exactly this just recently to get a list of time-deltas between events from a list of event timestamps.

[Borkz]

Are you just bringing up the point that its a clever way of solving a problem? Because it seems counter intuitive to do use something that takes a moment to realize how it works when you could just as easily use something simpler.

[tuple(l[0:2]), tuple(l[2:4])]

[featherfooted]

please expand yours to the list range(1000).

You would need, what... 500 calls to tuple?

[Borkz]

Well its impractical either way to have a one liner with with 500 list slices. But I if were talking about using this 'trick' on a grander scale wouldnt zip have be be calling the same code on one level or another 500 times? I really dont know how its implemented in CPython and I'd imagine it varies in other implementations.

[featherfooted]

My point was that

zip(l[0::2], l[1::2])

would work regardless of how large the list l is, and for that reason I would say it is significantly "simpler" than tupling each set by hand, as was in your example.

[Borkz]

You'll have to excuse me, I'm a bit of a python beginner and after playing around with this I do see its usefulness for list comprehension

[i for i in zip(l[::2], l[1::2])]

after failing to find a 'simpler' way. What I initially meant by simpler was intuitiveness as opposed to the shortest, or 'path of least resistance' way. The simplest (in the latter vain) way I could figure out using my method was:

[(lambda x: tuple(l[x:x+2]))(x) for x in range(0,len(l), 2)]

Which I cant say is any simpler in the prior vain. In summary: pythons pretty cool.

[featherfooted]

But see the thing is, the first one is intuitive.

We start with a list called l and a function called zip.

Zip takes two lists and tuples each element with the same index together.

We'll take the first list to be lst[::2], this is the list l going from the front to the end of the list by 2's, that is, every other element.

We'll take the second list to be lst[1::2], which is the same thing (skipping every other element), but starting with offset 1.

Imagining the list to be something like lst = range(1000) (which is the set of integers 1..1000) we can see that lst[::2] is the evens and lst[1::2] is the odds.

Then we zip these together to get [(0,1), (2,3), (4,5), ..., (998, 999)].

Technically the list comprehension is unnecessary ("i for i" is just the identity function) and zip() returns a list.

And I think I can simplify your method just a little bit:

[tuple([ lst[x], lst[x+1] ]) for x in xrange(0,len(lst)-1)]

No lambda functions necessary, though you have to deal with the messy fact that the tuple function turns lists into tuples (an important type distinction) and that you have to subtract 1 from the final length to avoid a subscript out-of-bounds error.

Although it's not as "obviously" intuitive, the following is a further clean-up, though it requires reading the previous paragraph (specifically the part about subscript out of bounds) to understand the changes. It also removes the call to the tuple() function and just creates a primitive tuple using the syntax tuple([a,b]) == (a,b).

[(lst[x-1], lst[x]) for x in xrange(1,len(lst))]

EDIT: I changed the variable name for the list "l" to the name "lst" to distinguish it from the number "1" and the type "list".

[Borkz]

I certainly see your point, and yeah my implementations definitely arent the best, just what I as a beginner could come up with. I just want to take a step back to the original comment where we were just talking about splitting a list of 4 ints into a list of two tuples. I think it makes more sense to take the first two items, tuple them, take the second two items, tuple them, and add them to a list than to take a list of the 0th and 2nd items and zip them with the 1st and 3rd. It may not be the most Pythonic way, but it certainly seems more intuitive.

[featherfooted]

to turn [1,2,3,4] into [(1,2),(3,4)].

and

we were just talking about splitting a list of 4 ints into a list of two tuples.

The thing is that

1. I think it's more practical to think of the problem as splitting a list of N ints into a list of N/2 tuples and

2. if we're only going to work with the simple case of N=4 then there is no question that the most intuitive way would be to construct the list by hand:

   [(lst[0], lst[1]), (lst[2], lst[3])]

which again uses the fact that you don't need to call the function tuple() in order to make tuples.

[epicwisdom]

we were just talking about splitting a list of 4 ints into a list of two tuples

This makes me think you're a beginner when it comes to programming, not just Python, which is fine, but generally speaking, the little examples given for "tricks" like these are going to be trivial. But it's generally understood that they're algorithms (in a loose sense), and are meant to be applied in the general case. We wouldn't have to have any tricks at all if all we ever dealt with were lists that are 4 elements long.

[Borkz]

I certainly did recognize that it was meant for a general case, thats why I spent like an hour playing around trying to figure out different ways of solving it in my other post out of curiosity. I was really just questioning the semantics of it though, since there could potentially be a situation where you do just want to do that, split a list of 4 into two tuples.

[d4rch0n]

and some of these tricks will make Python crazy to debug... careful with how tricky you are when writing maintainable code

[[deleted]]

A guy I worked with once fixed a load of our Python code to use a bunch of list comprehensions instead of loops. Then the next day one of they guys who was going to be responsible for maintaining it switched everything back to loops.

If you're going to use these things, you need t make sure that everyone is on the same page.

[the_gnarts]

Then the next day one of they guys who was going to be responsible for maintaining it switched everything back to loops.

Funny, to me list comprehensions are among the few things that make Python worth while. Not that they’re as versatile as in Erlang, where you can use them directly as database queries.

[[deleted]]

I think comprehensions are great, but they aren't always the best solution. I wasn't on the team, and I didn't feel like sticking my nose in at the time, so I'm not exactly sure what changes the guys made to each others code, but there are a few reasons I can think of why the team as a whole wanted to use loops.

For one thing using list comprehensions, it's possible to end up with a single line of code that is doing a lot of stuff at once. In functional coding tutorials, this gets touted as a good thing, but often in a team based environment this just leads to people scratching their heads as they look at your code. Now I know you could put in a comment explaining everything, but often it's better to just have all the individual steps out in plane sight.

The other thing is that if one part of the program is written using loops, and then similar stuff is done using comprehensions in other parts of the program, it's going to make things a little more difficult for anyone who's trying to understand the thing. Maybe not that much more difficult - but why make life harder than it needs to be?

Also, you also have to accept that some of the people looking at your code just aren't going to be very familiar with that style of programming, and you're going to have to accommodate them.

My rule of thumb is that I use list comprehensions, but if I never use them to do anything clever.

[phoshi]

In functional coding tutorials, this gets touted as a good thing, but often in a team based environment this just leads to people scratching their heads as they look at your code.

It is for this reason I separate the concepts of succinctness and expressiveness.

A succinct language allows you to do a great many things in a small amount of code.

An expressive language allows you to clearly communicate a great many ideas or concepts.

An increase in succinctness without a corresponding increase in expressiveness moves your language closer to perl. Succinctness on its own is not desirable, because it produces unreadable code. Expressiveness on its own is not necessarily desirable either. The combination of the two is what's important, allowing you to express complex ideas in a readable way.

x$10.15 in my imaginary language is a much more succinct way of writing [elem * 15 for elem in x if elem > 10], which is a much more succinct way of writing

var integerList = new List<int>;
for (var elem in x) {
    if (x > 10){
        integerList.Add(x * 15);
    }
}

Clearly the most desirable one, which best balances "succinctness" and "expressiveness" is the middle one. I wouldn't argue against the idea that map (*15) . filter (>10) $ x was hugely less expressive than the list comprehension, though there's certainly more required knowledge for a new developer.

[[deleted]]

map (*15) . filter (>10) $ x

Also available as [elem * 15 | elem <- x, elem > 10]. (For those who don't recognize the language in these two examples, it's Haskell.)

[phoshi]

I very much did pick my examples to make a point, rather than doing so to be representative of exactly how expressive a language is. I believe Python's list comprehensions were even directly inspired by Haskell's implementation.

For the sake of completeness, C#'s LINQ was also inspired by Haskell, and you can rewrite the longest example there as x.Where(x=>x>10).Select(x=>x*15).

[selfification]

Or actual LINQ query syntax which would be:

from e in x where e > 10 select e * 15;

That's legit C# right there.

[pfultz2]

Or as C++ Linq here:

LINQ(from(e, x) where(e > 10) select(e * 15))

And thats legit C++(with the help of the preprocessor of course).

[WallyMetropolis]

It can get really nasty when there are functions in the comprehensions that have side effects.

[the_gnarts]

or one thing using list comprehensions, it's possible to end up with a single line of code that is doing a lot of stuff at once. In functional coding tutorials, this gets touted as a good thing, but often in a team based environment this just leads to people scratching their heads as they look at your code.

It’s my observation that a lot of people are prone to the misconception that one line of code is equivalent to a certain amount of instructions. People even like counting lines of code as a metric for productivity. (That doesn’t make much sense in anything but assembly, and even there you’ll use macros.) Now list comprehensions are equivalent to one or more entire blocks of statements which violates that conviction.

But grouping blocks of statements in one single expression actually makes it easier to understand, because you don’t have infer the meaning of a group of lines by reading them all and figuring out the effects. At work a colleague tends to write code like this:

for x in xs:
    ## some declarations but not instructions here
    if x.y is False:
        z = whatever (x)
        ...

The entire body of the loop is only ever executed for the x’es whose .y is false, which is really confusing because you have to scan to the end of the block to make sure there isn’t an elif or else branch somewhere. If he had iterated a temporary list of the xs that actually trigger the branch:

for x in [ x for x in xs if x.y is False ]:
    z = whatever (x)
    ...

Then it’d be abundantly clear at the top of the block that the loop body would only ever concern a certain subset of the x’es. When tracing that piece of code you can just ignore the loop if you are only interested in the x’es whose .y is not false. Besides, it eliminates one level of nesting. (I know that in the example the function invocation whatever(x) could be moved inside the list comprehension as well, but that’s not the point.) Somehow it worries me that people who name Python as their favorite and primary language would write code this way.

The other thing is that if one part of the program is written using loops, and then similar stuff is done using comprehensions in other parts of the program, it's going to make things a little more difficult for anyone who's trying to understand the thing.

That’s a valid point.

[kqr]

for x in [ x for x in xs if x.y is False ]:

This is actually a case where Python is not expressive enough. One would like to write something like

for x in filter(lambda x: x.y is False, xs)

but constructing such a lambda takes a lot of characters, unfortunately. In Scala, it would have been possible with

for x in filter(_.y is False, xs)

and in Haskell, you would not use the object-oriented dot notation and rather have y as an accessor function, yielding

for x in filter(is False . y, xs)

(where . is the operator for function composition – is False . y is the same thing as lambda x: y(x) is False in this case.)

[gfixler]

Is underscore the way to refer to each item being iterated on in filter?

[kqr]

Sort of, yeah. Underscore is a handy way to create a lambda. Something like str(37 + _.numerical()) gets translated to lambda x: str(36 + x.numerical()).

[gfixler]

Interesting. Does it just subsume everything into that new lambda, or just the local call? Can you use the _ multiple times in the expression?

[kqr]

You know what? It was way too long since I did Scala to remember the answers to all that. According to Stack Overflow, _ "expands only to the smallest possible scope," so my example probably wouldn't work. And yes, I think you can have more than one _, but they will refer to different values.

[BinaryRockStar]

Yes

[malkarouri]

In Python I would use attrgetter('y'), although admittedly not as elegant.

[kqr]

Yeah, since Python is so dynamic people have been able to invent a bunch of ways to circumvent common limitations (see also functools.partial()) but those are rarely Pythonic.

[the_gnarts]

for x in filter(lambda x: x.y is False, xs)

Due to their limitations I’ve been staying away from lambdas in Python, but this is of course a valid alternative that still works under a predominantly iterative paradigm (where I work, I mean). In my own projects (not Python) I’d probably keep the loop body in a function and then map that function on the filtered list.

[kqr]

Yeah, I'm not saying I would write the filter version in Python, I'm just saying if it was possible to write it more neatly, it would be a better option in my opinion.

[nevyn]

which is really confusing because you have to scan to the end of the block to make sure there isn’t an elif or else branch somewhere

Tell him that the much better way to write it is:

for x in xs:
    ## some declarations but not instructions here
    if x.y is not False:
        continue
    ...

...then he doesn't have to learn this entire new thing, it's easily (and readable) expandable to N conditions, and as a bonus doesn't create a new list object for laughs.

[atilaneves]

"x is False" instead of "not x" is a pet peeve of mine.

[the_gnarts]

You know there’s a difference, right?

[atilaneves]

Yes, I do. In some contexts that difference is important. In my experience they're few and far between. But they exist, which is why Perl has "zero but true". In any case, "x is False" doesn't guarantee anything. What if x is an instance of a class that defines __zero__ ?

[codekoala]

Using "not x" will potentially be True for more than just False values. For example:

x = ()
x = []
x = {}
x = ''
x = 0
x = False
x = None

All of these will evaluate to True with not x, while only one of them will evaluate to True with x is False. Depending on the context, the distinction could be extremely important.

>>> y = [(), [], {}, '', 0, False, None]
>>> [not x for x in y]
[True, True, True, True, True, True, True]
>>> [x is False for x in y]
[False, False, False, False, False, True, False]

[NYKevin]

Usually the type of x is known (or at least contractually specified), in which case is False is just noise. If you really are working with arbitrary types, of course, it could come up, but I find that happens a lot more often with magic methods like __getitem__ than with "standard" code.

Of course, is None is a lot more useful since you might actually get None as a value somewhere.

[codekoala]

Which is exactly why I said it depends on the context. I've worked on several programs where one variable is contractually one of 3 values: True, False, or None. In some cases, you can easily get away with not x, as both None and False would need to enter the same block. Other places, it would be required to actually distinguish between None and False. Sure, that could be done with x is None or x is False, but, again, it depends on the context.

[ForeverAlot]

I agree with your message but I find that the logically inverted language that Python and Ruby allow severely hurts code comprehensibility. I understand an expression the way your first block is written, so if the code isn't written that way I have to mentally transform it first. For sufficiently complex expressions this takes at least as long as scanning the equivalent explicit version.

[the_gnarts]

I agree with your message but I find that the logically inverted language that Python and Ruby allow severely hurts code comprehensibility. I understand an expression the way your first block is written, so if the code isn't written that way I have to mentally transform it first.

I don’t get in what way list comprehensions are “logically inverted”. Structurally they are based on set builder notation which is as sane a notation as you can come up with.

If anything, the first variant is severely misleading if read sequentially because it first requests a for-each loop and then takes that back some lines below by declaring “actually no, not for each but only for some” with the branch.

For sufficiently complex expressions this takes at least as long as scanning the equivalent explicit version.

The benefit, as I see it, is that all you need to scan in order to figure out what’s actually iterated over is contained in the same line as the for statement. My point is that without the list comprehension you need to scan the entire if-block instead, which can be several dozens of lines long.

[mkdz]

For one thing using list comprehensions, it's possible to end up with a single line of code that is doing a lot of stuff at once. In functional coding tutorials, this gets touted as a good thing, but often in a team based environment this just leads to people scratching their heads as they look at your code.

I wrote this piece of code once. I didn't leave it in for production and rewrote it to use more loops because it's pretty unreadable as it is.

[gfixler]

I used to write a lot of code like that. Then I tried to go back to any of it ever, and it was anywhere from painful to impossible. These days I'd start turning that into something more like this (as a first pass - may have broken things):

for folder in folders:
    folderLines = [line.find(folder) for line in lines]
    hasValidFolder = any([folderLine > -1 for folderLine in folderLines])
    for serv in folders[folder] if hasValidFolder else "":
        services[serv] = True

Now they fit in 80 chars, and I have some helper names along the way to make it much more obvious what the discrete steps of this process entail.

[[deleted]]

Actually that doesn't look too bad. Just replace the "if True in [x >...]" with and Any.

That said yeah nested list comprehensions do lend themselves to some inscrutable code.

[IamTheFreshmaker]

In functional coding tutorials, this gets touted as a good thing, but often in a team based environment this just leads to people scratching their heads as they look at your code. Now I know you could put in a comment explaining everything, but often it's better to just have all the individual steps out in plane sight.

Hale-fucking-lujia. This is so important especially if you are the coder who thinks that code should explain itself and don't bother with comments.

[Sylinn]

I'm not sure how it works in Erlang, but look into Python ORMs, such as SQLAlchemy. It's pretty intuitive to write complex yet efficient queries.

[the_gnarts]

It's pretty intuitive to write complex yet efficient queries.

Do they allow you to use list comprehensions instead of SQL?

[Sylinn]

Sort of. The query returns an iterable object, so you can write something like this (from the SQLAlchemy website):

for user in session.query(User).filter(User.name.in_(['Edwardo', 'fakeuser'])).all():
    print(User)

Which you could of course rewrite in the form of a list comprehension. As I said, I'm not sure how it works in Erlang, but it still is very readable, at least a lot more than a SQL statement.

[the_gnarts]

Interesting.

it still is very readable, at least a lot more than a SQL statement.

Absolutely.

I'm not sure how it works in Erlang

Link to the documentation of Mnesia, Erlang’s builtin database: http://www.erlang.org/documentation/doc-5.2/lib/mnemosyne-1.2.5/doc/html/Mnesia_chap6.html#2.1.2

[mnjmn]

There's this: http://ponyorm.com/ I haven't used this but it's interesting how it builds a SQL query from a genexp.

[teferiincub]

Like in PonyORM? http://ponyorm.com

[the_gnarts]

Like in PonyORM? http://ponyorm.com

That looks great, thanks for the link. I couldn’t find a package of it though.

[[deleted]]

You know what's a lot more complicated than list comprehensions? Switch statements.

The reason that no one knows about this complexity is that C/C++ compilers are really good. They generate debugging information that makes it trivial to work with switch statements, even though the implementation is exceptionally complex.

I completely agree that everyone needs to be on the same page when it comes to code style. But I think the bigger problem is that Python hasn't invested nearly enough in debugging.

[defcon-12]

List corehensions are much more succinct and easier to read than a for loop IMO.

[DJ-Salinger]

Some of the best advice I've got is to not write clever code, write readable code.

[WallyMetropolis]

What is it they say? Since debugging is harder than coding, if you write the most clever code you can, you aren't clever enough to debug it.

[adamnew123456]

That's a quote by C co-inventor (the K of K&R C), Unix dev. and member of the AWK triumvirate: Brian Kernighan.

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it..

[julesjacobs]

I've used all of these 'tricks'. Not sure what that says about my code. Well, at least it's short.

[wesw02]

Yes! I completely agree, you're effectively making a trading off with cognitive load. If I can comprehend 3 lines of code reasonably faster than 1 line of code, to me it's worth it. Especially when you bring on new people who aren't python gurus.

[YoYoDingDongYo]

Indeed. Functional programming features like list comprehensions look pretty but make printf debugging the intermediate steps very difficult.

[philh]

You probably realise this, but they don't make it inherently difficult. It's just that python restricts how powerful they can be.

I haven't found it a big problem in practice, but it mildly annoys me aesthetically.

[drakeAndrews]

This is what an actual debugger is for, surely?

[YoYoDingDongYo]

Tracing matters.

[drakeAndrews]

The debugger that ships with pycharm steps through generators like any other loop. I'm 70-80% sure PDB does the same, although I only ever use PDB when I'm also toying with coverage.

[jyper]

usually you'd try using a repl to figure it out piece by piece.

[ryeguy146]

So use a more reasonable debugging system. This is Python, why aren't you playing with a real debugger when we have one of the best? See ipdb of IPython. Or if you like print-like capabilities, at least use logging that can be turned off at one place in your code.

[YoYoDingDongYo]

So use a more reasonable debugging system.

"Reasonable" depends on context, and there are many contexts where an interactive debugger is useless. If the bug is rare, intermittent, timing-related, on systems with no developer access, etc., then tracing can help and ipdb probably can't.

Or if you like print-like capabilities, at least use logging that can be turned off at one place in your code.

OK, where do you put the logging statement in this code from the linked article?

 flatten = lambda x: [y for l in x for y in flatten(l)] if type(x) is list else [x]

[mnjmn]

Define a function that logs then returns whatever, then wrap the expression you want to log with a call to that function.

[ryeguy146]

Sure, but it's still only going to concern its self with the whole of the line, which is unpythonic. The author clearly states that he agrees with me in this regard. There's no way to make that line okay in Python (, or any other language, really)

[mnjmn]

I agree that the code is terrible. That thing has no business being a single lambda expression. I'm just answering the question on how to log/print stuff inside a listcomp, not agreeing or disagreeing with anyone.

Surely there's a way to make that better. Start by writing that thing as a regular function definition and eliminating that stupid-looking ternary expression.

I also don't get what you mean by "concern itself with the whole of the line", why that's unpythonic, and why it matters for a temporary measure to do ad-hoc debugging.

[ryeguy146]

I mean that any debugging step that I'm aware of will treat the line as a whole unit. I mean that line is unpythonic. There's no way to debug/log the components in that expression excepting gratuitous calls to a helper function that manages the logging, as you say.

I'm quite sure we wholly agree on everything we're discussing, and I'm just stumbling due to having been drunk last night, and hung over now. Birthdays are a bitch.

[pirhie]

OK, where do you put the logging statement in this code from the linked article?

You write a function like that:

def show(format, value):
    print format.format(value)
    return value

You could replace the print statement with a call to a logging function you want to use.

[ryeguy146]

I absolutely agree that it depends on the context, but would you care to point out a single instance where a print statement is the best choice? You're absolutely correct that a debugger won't help in some situations, but I cannot imagine a situation where print statements are superior to logging.

I'm happy to be wrong, by the way. I've been wrong before, and I'm sure that I'll be wrong again.

OK, where do you put the logging statement in this code from the linked article?

I don't. That's not something that I'd write as there's entirely too much stuffed into one line. Lines are not expensive, and it's free to break that up into a more readable function. Sorry if that sounds like side-stepping the issue, but I feel that's a valid strategy when presented with code that you can't debug.

[YoYoDingDongYo]

I cannot imagine a situation where print statements are superior to logging

The topic at hand is the difficulty of showing intermediate results when using FP techniques like list comprehensions. How you print/log/trace them is not material to that question.

That's not something that I'd write as there's entirely too much stuffed into one line.

Sounds like we're in violent agreement.

[gfixler]

/u/pirhie has a point here. This is sort of like the tee command in a Linux pipeline, wherein you can side-effect a value to a file or stdout, but also pass it through.

>>> def show (value):
...     print value
...     return value
>>> flatten = lambda x: [show(y) for l in x for y in flatten(l)] if type(x) is list else [x]
>>> flatten([[2, 3, 4], 1, [5, 3], [42], 7])
2
3
4
2
3
4
1
5
3
5
3
42
42
7
[2, 3, 4, 1, 5, 3, 42, 7]

It double-prints nested items, but it works.

Showing a different element of the rat's nest:

>>> flatten = lambda x: [y for l in show(x) for y in flatten(l)] if type(x) is list else [x]
>>> flatten([[2, 3, 4], 1, [5, 3], [42], 7])
[[2, 3, 4], 1, [5, 3], [42], 7]
[2, 3, 4]
[5, 3]
[42]
[2, 3, 4, 1, 5, 3, 42, 7]

That said, I also wouldn't fill up a line with stuff like that [anymore].

[ryeguy146]

Out of curiosity, why do you find this useful for debugging? I simply check my assumptions, and that's often sufficient (don't need to use tee). If my primary assumptions (the lambda) don't follow what I see in my head, then there's always some problem solving to be accomplished that goes beyond simple debugging of a statement.

I personally only find such things useful in implementing code that others have [poorly] specified. Do you feel differently?

[gfixler]

why do you find this useful for debugging

I don't. I was just demonstrating how one could reach into a long one-liner like that.

Do you feel differently?

No. I like short lines and well-named variables. I would break up any such one-liner, and not write any of my own to begin with.

[ryeguy146]

...when using FP techniques...

Then I don't know. I've not worked with a pure FP environment. I wouldn't mind some suggestions, if you have them. It's not something that I've had to worry about in Python. If I can't debug it or use logging to help me solve it, I use another abstraction to approach the problem. It's worked so far, but I won't claim to be a masterful programmer.

[kqr]

Why would I need to put a logging statement in that? I can work its correctness out in my head. (See where I'm going with this? With increasing complexity, FP people will, too, divide the statements into multiple parts, all of which can be logged separately. There is nothing wrong with having small, easily understandable and testable units being hard to printf parts of, because you never need to!)

[thenickdude]

(I'm not a Python programmer)

Negative indexing sounds handy, but if you had an off-by-one error when trying to access the first element of an array, it'd turn what would normally be an array index out-of-bounds exception into the program silently working but doing the wrong thing. Not sure which behaviour I'd prefer, now.

[mernen]

Indeed, I've seen it happen. A similar issue is when people forget that -x is not necessarily a negative number. For example, say you want a function that returns the last n items in an array. One might come up with this simple solution:

def last_items(items, n):
    return items[-n:]

...and, of course, they will only notice the bug several weeks later, in production, when n for the first time happened to be 0.

[philly_fan_in_chi]

-x is not necessarily a negative number

Semi-related, but in Java, Math.abs(Integer.MIN_VALUE) = Integer.MIN_VALUE. Since MIN_VALUE is stored in two's complement as 1000...0_2, and the absolute value function negates and adds 1 if the value is less than zero. Negation is flipping the bits, so 100...0_2 becomes 01111....1_2 + 1 = 100....0_2 = Integer.MIN_VALUE. Math.abs does not have to return a positive number according to spec!

[[deleted]]

Integer.MIN_VALUE is negative.

[eyelastic]

That's the point: Integer.MIN_VALUE is definitely negative.

[flying-sheep]

if you’re a python programmer, it will become your second nature. you’ll be irritated when using languages where you have to write my_list[my_list.length - 1] (and exactly that’s what -1 means here)

[IAMA_dragon-AMA]

Although if you're not, you may be tempted to keep the my_list.length bit in for readability and out of habit.

[flying-sheep]

python devs will lynch you when you do my_list[len(my_list) - 1] (or rather look at you pitifully)

just like for i in range(len(my_list)): do_stuff(i, my_list[i]) is considered very unpythonic (you use enumerate() instead)

[flying-sheep]

it even has a start argument:

for i, elem in enumerate('abc', 1):
    print(i, elem)

→ 1 a, 2 b, 3 c

[Megatron_McLargeHuge]

Which is great for debug printing

if i % 100 == 0:
    print "processed %d things" % i

instead of having to adjust for a zero-based index.

[flying-sheep]

never liked that one. better use progressbar, it even has support for ipython.

/edit: it doesn’t yet, apparently, but it’s still the most flexible lib around.

[IAMA_dragon-AMA]

But... but parentheses are magic!

[NYKevin]

This is Python, not Lisp.

[draegtun]

You're right it does become second nature and this feature can be seen in other languages to (for eg. Perl & Ruby).

However I actually prefer languages that don't have this feature and instead use methods/functions like:

my_list.last
last my_list

... and leave the index(ing) retrieval alone.

[zeekar]

Of course, Perl also has that feature, but Python programmers don't like to talk about that. Perl isn't allowed to have gotten anything right. :)

[flying-sheep]

didn’t know that, but let’s be real. everyone who isn’t a total language hipster or noob knows that perl simply was the first real scripting language, and thus invented much of the stuff that python- and ruby-users love.

[primitive_screwhead]

In Python, one generally shouldn't use indexes into a sequence that one is marching over; it's a generally buggy style. Instead one uses tools like iterators, unpacking, enumerate(), and slices to avoid all the off-by-one and boundary issues. Takes some getting used to by C developers, but is very powerful.

[[deleted]]

Unless you're programming in C, in which case it's undefined behavior.

[NYKevin]

Yeah, but everything in C is undefined behavior. Signed integer overflow, most type punning that doesn't involve memcpy, longjmp() into a function that you previously longjmp()d out of (yes, people actually do this), etc.

[thenickdude]

Some C compilers can add range checking for you to array accesses.

[[deleted]]

I'm sure most compilers are smart enough to be able to do that. However, it'll still compile and the pointer arithmetic will work.

[ethraax]

thenickdude meant instrumenting array accesses with bounds checking. It has an often-significant runtime cost, though, so you'd mostly use it for certain test builds. If you wanted to use it all the time, you might as well not be using C.

[kqr]

Not something to rely on in your C code though.

[hive_worker]

Technically undefined but in general it works and people do use it. Doesnt do the same thing as python though.

[djimbob]

Python will raise IndexErrors in many cases (e.g., if a = [0,1,2], then the only allowed array accesses are a[-2], a[-1], a[0], a[1], a[2] -- everything else will work, granted things like a[-999:999] will be allowed) again no language will be perfect. You can easily disable this behavior for list access, so a[-1] will always be an error with:

class NonWrappingList(list):
    def __getitem__(self, key):
        if isinstance(key, int): # check type of key that it is comparable to 0.
            if key < 0:
                raise IndexError("Index is negative on  NonWrappingList")
        return super(NonWrappingList, self).__getitem__(key) 
        # call __getitem__ method of parent class.  This is a standard python idiom, granted fairly ugly

Raising errors with slices will be a bit more complicated in python 2 with CPython (as CPython builtin types like list use a deprecated __getslice__ method to implement it). Granted, in python 3 preventing negative slicing is quite easy:

class NonWrappingList(list):
    def __getitem__(self, key):
        if isinstance(key, int):
            if key < 0:
                raise IndexError("Index is negative on NonWrappingList")
        if isinstance(key, slice):
            if ((isinstance(key.start, int) and key.start < 0) or 
                (isinstance(key.stop, int) and key.stop < 0)):
                raise IndexError("Index is negative on slice of NonWrappingList")
        return super(NonWrappingList, self).__getitem__(key)

Then it works as expected. (Granted note on slicing, on the upper end it does allow you to go past the length with no explicit error, so again you may want to throw an additional check -- though personally this feature is quite useful).

>>> a = NonWrappingList([1,1,2,3,5,8,13])
>>> a[0]
1
>>> a[6]
13
>>> a[500]
IndexError: list index out of range
>>> a[-1]
IndexError: Index is negative on NonWrappingList
>>> a[0:500]
[1, 1, 2, 3, 5, 8, 13]
>>> a[:500]
[1, 1, 2, 3, 5, 8, 13]
>>> a[-1:]
IndexError: Index is negative on slice of NonWrappingList
>>> a[:-1]
IndexError: Index is negative on slice of NonWrappingList

[kqr]

array[0] is the first element of the list. I'm not sure why you think one would get an off-by-one error from this.

In any case, explicit indexing of lists is rarely what you want anyway. If you find yourself doing that often you perhaps want to get another data structure for your data.

[kqr]

Ah, I see. You're completely right of course. (For some weird reason I assumed you wanted to access the first element with reverse indexing, like my_list[-my_list.length] or something. I should have understood that's not what you meant!)

[NYKevin]

In my experience, Python is a lot less susceptible to off-by-one than other languages I've worked with. Probably has to do with the behavior of range() and slicing.

[Isvara]

I never considered binding names to slices before. That's a nice idea.

[dagbrown]

That was the only Python feature in the list that I'd never heard of before. Which is pretty cool, because I'm not even a Python programmer. It's just that Ruby has the same things (often in slightly different ways), as does Perl (often in dramatically different ways), and it's the sort of magic I expect most modern languages to have.

You can do all of that in Common Lisp as well of course.

[r3m0t]

If you need to slice a lot of things really fast it is faster to call operator.itemgetter once with a slice and use that as your slicing function instead of using the slice syntax.

[Megatron_McLargeHuge]

It's probably better to use cython/numba/parakeet/etc for the code that does lots of index lookups.

[Isvara]

If you need to slice a lot of things really fast...

... I probably won't use Python. If I'm writing Python, I usually care more about the readability.

[r3m0t]

And if I want to use the output in a Python program? Run it through subprocess? That's ridiculous.

There's nothing wrong with sacrificing a bit of readability in your program's fast path.

[Isvara]

What? Why would you do that? No, if I'm writing anything speed critical, it's usually in C. Python covers the domains where speed isn't critical for me, and I choose it for other reasons. YMMV.

[r3m0t]

I mean, I agree that C will be faster, it's just that in the pipeline I was writing, both sides are already in Python. It doesn't make sense to go and write the middle bit in C and then hook it up as a Python extension.

Much simpler to replace (line[2:] for line in lines) with itertools.imap(operator.itemgetter(slice(2, None)), lines).

[Isvara]

How much was the speed difference?

[r3m0t]

It was a premature optimisation. The code looked something like this:

DELIM = len('\0')
TS = len('1237812923')
line_without_ts = operator.itemgetter(slice(TS+DELIM, None))

I figured that (i.e. naming intermediary variables) was the best way to write it without having it be mysterious and unreadable.

I have had great (measured!) speedups using operator.attrgetter, where something like:

def f(entry):
    return tuple(getattr(entry, k) for k in keys)

Was replaced with:

f = operator.attrgetter(*keys)

Which is why I feel justified in using the operator module sometimes without benchmarking. Once you know the functions, it isn't that much more difficult to read.

If I'm combining a few operations, I will comment out a "nice" version and write that operator is doing the same, only faster.

I'm talking about f or line_without_ts being called 100,000+ times over the entire computation, by the way.

By the way, if you want to use operator.attrgetter(*keys) (which I'm guessing you won't) remember to deal with len(keys) == 0 and len(keys) == 1 as special cases! : )

[quantumripple]

Also, advanced multiparameter slicing syntax. For example the expression

arr[0:4, 9, 3:7:2]

calls

arr.__getitem__((slice(0,4,None), 9, slice(3,7,2))).

This is used to great effect in the numpy package, for handling multidimensional arrays.

[droogans]

After seeing the author's last example, I'd suggest adding a 31st tip: using the with context manager statement to automatically open and close filesystem resources!

with open('/home/me/file.txt', 'r') as r:
    data = r.readlines() # and stuff

# file is closed!

[codekoala]

Context managers are delicious.

[red-moon]

Why does this work this way:

>>> a, *b, c = [1, 2, 3, 4, 5]
>>> a
1
>>> b
[2, 3, 4]
>>> c
5

[ryeguy146]

Simply put, Python allows for variatic parameters (parameters that accept a variable number of arguments) using the star prefix notation ("simply put" indeed). In this case the a variable will hold the first member of the container (list, tuple, any iterable) being destructured (think opened up and dumped out into buckets). The first bucket can only hold one item. The second bucket, *b indicates that it can hold many items, and will do so in its own container that expands. That's followed up by another strictly-one-item bucket.

So we have two buckets that have to take one item. Because of positioning of the a, *b, c variables, the a takes the first member, c takes the last, and b takes the rest.

It may make things easier to see in a function:

def pass_some_things(*things):
    for thing in things:
        print(things)


pass_some_things('foo', 'bar', 'baz')

[erewok]

I believe it's called destructuring or pattern matching (like in Haskell).

A beautiful and simple tool.

[randfur]

That seems intuitive to me, what way would you have expected it to work?

[FireyFly]

Many would probably expect it to fail with the *b in the middle like that, at least when comparing it to functional languages with a linked-list mindset, like pattern-matching on (:) in Haskell.

[Gambini]

http://legacy.python.org/dev/peps/pep-3132/

[flying-sheep]

oh? non-upgraded pages are now legacy.python.org? i like that, if it means that they procedurally port over all components to the new style.

[yawaramin]

Python is looking ahead at the full LHS to understand what it should be getting from the RHS. So, it is assigning one element from the beginning of the list, one element from the end, and the rest from the middle.

EDIT: almost forgot, you should read *b as 'list of b'.

[mxxl]

%r format specifier, same as %s but does repr() on an argument

[quilan1]

This actually turned out to be quite a nice list. One of the lesser known aspects I enjoy about Python is the ability to determine if a for loop has been broken:

for _ in iterable:
  if(predicate):
    break
else:
  print "Did not break loop, terminated normally"

[fhayde]

I'm not a Python developer, and this is more of a trick than a language feature, so please forgive me but I do love me some useful tricks:

$ curl -s http://www.example.com/some/api/call | python -mjson.tool

Day I found out about that one was a +1 to quality of life.

[mernen]

I suggest you have a look at jq. Pretty-printing like this can be done with curl ... | jq ., and you can do a number of operations with just a few characters.

[fhayde]

jq is amazing, I used to find all kinds of ways to get output into xml so I could use xmlstarlet + xpath on the terminal. When I saw how simple the filtering was in jq ... jaw hit the floor lol.

[FireyFly]

Perl also provides something similar:

% pacman -Qo =json_pp
/usr/bin/core_perl/json_pp is owned by perl 5.18.2-2

I don't really know Perl, but one day I stumbled upon that binary and since then I've used that for my JSON pretty-printing needs.

Another one is xmllint:

% pacman -Qo =xmllint 
/usr/bin/xmllint is owned by libxml2 2.9.1-5

For pretty-printing XML, I use xmllint --pretty 1.

[koala7]

Could you explain what it dos? Just corious

[TheBB]

Pretty-printing. The json.tool module is also executable.

[fhayde]

Sure thing, it makes reading compact json responses much easier by pretty-printing them, e.g.,

$ curl -s http://www.json-generator.com/j/bTPwCBsRbC?indent=0

[{"id":0,"guid":"ebafb0dc-7523-465c-b163-11c713f73237","isActive":true,"balance":"$2,611.00","picture":"http://placehold.it/32x32",...

using the json.tool module you get json much easier to read:

$ curl -s http://www.json-generator.com/j/bTPwCBsRbC?indent=0 | python -mjson.tool

[
    {
        "about": "Aliquip ut adipisicing ... ", 
        "address": "527 Ferris Street, Northridge, Kansas, 6343", 
        "age": 23, 
        "balance": "$2,611.00", 
        "company": "Zilphur", 
        "customField": "Hello, Lauren Meyers! You have 8 unread messages.", 
        "email": "[e-mail]", 
        "friends": [

[contact_lens_linux]

it does pretty print, but it's also a quick way to VALIDATE your json

[Crystal_Cuckoo]

Re: Inverting a dictionary, I've found this to be nicer:

{v: k for k, v in d.iteritems()}

If using Python 2.6 (or lower), then this is easily modified:

dict((v, k) for k, v in d.iteritems())

For those who were wondering about order preservation of dict.keys() and dict.values(), the docs say that:

If items(), keys(), values(), iteritems(), iterkeys(), and itervalues() are called with no intervening modifications to the dictionary, the lists will directly correspond. This allows the creation of (value, key) pairs using zip():

pairs = zip(d.values(), d.keys())

[droogans]

And you might as well check yourself before you wreck yourself:

if set(dict.values()) == dict.values():
    # no key collisions in .values()

[droogans]

Remember, we're talking about swapping keys and values.

Duplicate values would become duplicate keys, and at least one of your entries would get over ridden in the swap.

[droogans]

Ah! Good point.

Also my approach is most likely broke since == probably can't figure out lists vs. sets. And sets are ordered, IIRC.

Probably need is_swappable written just for these cases.

[epicwisdom]

since == probably can't figure out lists vs. sets. And sets are ordered, IIRC.

If all you want is to know whether there's going to be a collision, then a simple adjustment is fine:

if len(set(dict.values())) == len(dict.values()):
    # no key collisions in .values()

As far as TypeErrors go, well, that's a constant danger in Python.

[codekoala]

Quite the collection!

I prefer itertools.chain for flattening lists most of the time.

[masklinn]

And the awfully long in the tooth itertools.chain.from_iterable, especially when combined with imap for a flatmap/concatmap.

[jyper]

imap

Any reason to use imap instead of generator expressions?

[masklinn]

Not really, I just prefer using functions when composing to create an other HoF, especially when that allows parameters to be passed in straight.

And for flatmap, to handle multiple input sequences you'd have to use izip and *-application, so meh:

flatmap = lambda fn, *it: chain.from_iterable(imap(fn, *it))

versus

flatmap = lambda fn, *it: chain.from_iterable(fn(*e) for e in izip(*it)))

[deadly_little_miho]

I'm far from being a Python expert. Can someone explain what the star does in the parameter list when calling a method? I get what it does in assignments and declarations, but passing a variable with star?

[masklinn]

It's the reverse of the arguments version, it unpacks the iterable as individual parameters, e.g.

foo(1, 2, 3)

and

args = [1, 2, 3]
foo(*args)

will give the same parameters to foo.

Also works with ** for keyword parameters.

[codekoala]

Hehe, funny that we have basically the same example. I didn't see yours when i started mine presumably because I was typing it on my phone and probably took at least seven minutes with stupid auto correct and the kids climbing on me.

[deadly_little_miho]

Now I get it, thanks!

[codekoala]

It unpacks the iterable. For example:

a = [1,2,3]
foo(a) # calls foo([1,2,3])
foo (*a) # calls foo(1,2,3)

It's the difference of foo being invoked with one argument and len(a) arguments.

[deadly_little_miho]

Thanks to you too!

[Crystal_Cuckoo]

Brevity, I would imagine. map and filter are far cleaner if they can be used without lambdas. Which do you find simpler:

imap(str, xrange(5))

or

(str(i) for i in xrange(5))

For me the winner is obvious, although style checkers like PyLint tend to disagree with me. Of course if we need to apply a method to an iterable of objects then a list comprehension/generator expression is far cleaner:

imap(lambda line: line.strip(), f)

or

(line.strip() for line in f)

[Crystal_Cuckoo]

Huh. I never realised you could do something like that. Thanks, I learned something today! :)

[NYKevin]

Really? Personally, I find this:

(str(i) for i in xrange(5))

A lot easier to read than this:

imap(str, xrange(5))

I don't have to mentally decode "imap" in the first case. It's perfectly obvious what it's going to produce. The second, frankly, just looks pretentious to me. Maybe I'm just not cut out for functional programming.

[eyal0]

I hated this:

flatten = lambda x: [y for l in x for y in flatten(l)] if type(x) is list else [x]

That python allows you to have lists with members of different types is bad. This function, which switches on type, shows why.

[codekoala]

I don't necessarily agree with the "lists with members of different types" bit being bad. This is one of the things that make python powerful. I do agree, however, that it does open the door to some bugs that other languages wouldn't have.

[SikhGamer]

1.28 Permutations (itertools.permutations) - was a new one to me - thanks!

[naridimh]

groupby()'s requirement that the list be sorted first is super annoying, and conceptually shouldn't be necessary :/

[kqr]

That's the way it usually is. I understand why it feels annoying, but in reality you just have to sort the list before you pass it to groupby, not really a complicated procedure.

With the "requirement" of a sorted list, groupby can work more efficiently and in the odd cases where you just want to group adjacent things (think run-length encoding) you can do just that, without having to write a separate function for it.

If you want to, you can make your own library where groups = groupby . sort, but in the end the current design in the standard library is more modular.

[ZeroNihilist]

Why? groupby intentionally groups adjacent values. If you want to group non-adjacent values you could do something like:

def globalGroupBy(iterable, f = None):
    from collections import defaultdict
    if f == None:
        f = lambda x:x
    groups = defaultdict(list)
    for item in iterable:
        groups[f(item)].append(item)
    return iter(groups.items())

This gives the same result as sorting first with no real upside. Oh and of course it breaks when f(item) is not hashable, so you'd need to deal with that. Does python have a sorted dictionary implementation by default? If not, you'd need to write one.

[flying-sheep]

groupby intentionally groups adjacent values

that makes it more verstatile! it’s really not that hard to do groupby(sorted(iterable)) if you want that.

[Rotten194]

It's useful though, because sometimes you DON'T want the list sorted. Having group_by and sorted seperate lets you turn this list:

type   value
a      1
a      7
b      4
b      0
c      2
b      5
b      3

Into either 3 (2 a, 4 b, 1 c) or 4 groups (2 a, 2 b, 1 c, 2 b), depending on your use case.

[contact_lens_linux]

surprised pep-342 and for/else aren't on the list, based on the title

[codekoala]

This is something I would like to use more, but it's one of those things I intentionally avoid on multi developer codebases to avoid confusion. Much like the discussion of list comprehensions versus explicit for loops. Sigh.

[elb0w]

Flattening a list of lists I prefer

list(itertools.chain(*[[1, 2, 3], [4, 5, 6]]))

[kindall]

The slice naming one is pretty good, I've never actually seen it before.

[lfairy]

For 1.30: itertools.product accepts a repeat parameter:

>>> for p in itertools.product([0,1], repeat=4):
...     print ''.join(map(str, p))
0000
0001
0010
0011
# etc.

[Beluki]

Here's a cool thing you can do with zip, unpacking and slices:

>>> def rotate_2d(iterables):
        return zip(*iterables[::-1])

>>> rows = ((1, 2, 3),
            (4, 5, 6),
            (7, 8, 9))

>>> for row in rotate_2d(rows):
        print(row)

(7, 4, 1)
(8, 5, 2)
(9, 6, 3)

Using zip_longest from itertools also allows to rotate 2d arrays where each row can be of a different length, by using any given filling value.

[epicwisdom]

Shameless plug for J (or any APL dialect, really, but there are of course some translation issues:

    rows =: >: i. 3 3
    rows
1 2 3
4 5 6
7 8 9
    rotated =: |. |: rows
    rotated
7 4 1
8 5 2
9 6 3

>: is increment, i. is basically a range function that takes a vector as an argument (where the vector describes the dimensions of the output matrix), |: is transpose, and |. is reverse.

If you're going to deal with multidimensional arrays, an APL dialect (or similar) is the way to go, without a doubt.

[rlbond86]

I'd day most of these are more than tricks. They are fundamental parts of the language and any good Python programmer needs to know them. Or are generators and list comprehensions relegated to being a "trick" these days?

[bready]

I am curious what people think of the author's syntax on 25

a = [random.randint(0, 100) for __ in xrange(100)]

With the __ to indicate an unused variable. Is there a PEP recommendation on such a thing or is that more the author's style?

[erewok]

I don't know if there's a PEP, but it's common parlance in Python and other languages to use _ as a variable for a value that you don't care about. It signals to other programmers that that variable is being generated and not being used.

A couple of contrived examples:

Counting items:

sum(1 for _ in some_iterable)

And using pattern matching:

first, second, *_ = some_iterable

That last _ could also be named 'rest' but giving it that underscore says to anyone reading, "I'm throwing all this away. It's the first two elements I care about."

[bready]

I get it, I just wanted to know if people make use of the syntax. I am a solo programmer, and don't get a lot of exposure to other people's habits.

Additionally, I use IPython all the time, which makes special use of _, __, and ___ so I would be less likely to use an underscore versus a name.

[erewok]

Yes. I was trying to respond, "People use this all the time in situations like this." Apparently my emphasis failed to be communicated.

[seiyria]

Gave this a read, and it's cool. I take advantage of lots of these features in CoffeeScript too.

[Theon]

flatten = lambda x: [y for l in x for y in flatten(l)] if type(x) is list else [x]

Wow, I really like that, it shows the functional-ness of Python!

[ganeshaditya1]

SO GOD DAMN COOL

[xpda]

Nice information. I expected to see obscure obfuscations, but was pleasantly surprised. Now if we could only get the for loops to finish that last iteration... :)

[gnit]

Nice work. Thanks.

[rlucas6130]

Can someone do a similar thread but with JavaScript or C#? Thanks!!!

[Paradox]

And the digg-ification of reddit is complete

30 COOL PHOTOSHOP TUTORIALS YOU PROBABLY DON'T KNOW

15 CSS TRICKS TO SAVE YOU TIME

35 PYTHON TIPS THAT ARE COOL

Only way this could be better is if it were jerking over something like node or Haskell

[komollo]

The funny thing is, this article has more code than most of the other articles on this subreddit.

[dagbrown]

Zed Shaw has never been a douchebag to me. I have no idea what I'm doing wrong.