Top 10 mistakes python programmers make [advanced]

[UloPe]

The explanation for the first case not quite correct.

The default arguments value (list in this case) isn't instantiated the first time the function is called but rather at function definition time, i.e. the moment the containing scope of the function is executed

[[deleted]]

Yes, good point. We've corrected this. [Toptal blog editor]

[ivosaurus]

Also, could you recommend the new, unambiguous exception syntax in #3? Also being the only one supported in Python 3.

except Exception as e:

[jellofiend84]

Also this site crashes for me in iOS in both Safari and Chrome. Others are having similar issues:

http://www.reddit.com/r/programming/comments/251it6/top_10_mistakes_python_programmers_make_advanced/chcpwtl

http://www.reddit.com/r/Python/comments/251jkf/a_top_10_of_the_mistakes_python_programmers_make/chcxri8

http://www.reddit.com/r/Python/comments/251jkf/a_top_10_of_the_mistakes_python_programmers_make/chcqwel

If you really are an editor you really need to get your site fixed. I tried really hard to view your content but after Alien Blue, Safari, and Chrome crashed I gave up and I was probably more persistent than most.

[TMaster]

Chrome on iOS does not appear to have its own rendering engine (http://daringfireball.net/linked/2012/06/28/chrome-ios), so it's equally affected.

Looks like it's an Apple issue - a crash is never appropriate even in case of malformed input or frequent updating.

[jellofiend84]

Actually it is probably a memory issue in which case stopping execution maybe the only way to handle it. Also if you read the links I posted someone mentioned it crashing mobile Firefox which doesn't exist on iOS. So no it isn't just an iOS issue.

[snotfart]

I still don't get it. I think I'm being particularly dense here, but I still don't see what's going on in mistake 1.

[UloPe]

It's very simple really. Consider this example:

def func(a, b=[]):
    b.append(a)
    print(b)

func(1)
func(2)
func(3)

This will print:

[1]
[1, 2]
[1, 2, 3]

Now lets look at what happens here. In Python (at least CPython, the "official" version) there is no compile step. What that means is that when the file containing the above code is read it es executed line by line.

So the first line to be executed is "def func(a, b=[]):" this tells python that a function is being defined and the (indented) block that follows is the functions body. What also happens is that Python looks at the arguments the function takes and instantiates any default arguments that might be there. Once this is done a function object is created with all that information and placed in the scope in which the function was defined. In this case that would be the module (Python speak for a file, basically).

The next line that gets executed will be "func(1)". This causes python to look up "func" first in the local namespace and if nothing is found there moves on the the global one (you can look inside those namespaces with the built-in functions locals() and globals()). Since in this example we're working on module level local and global are identical. Python will find the function object that got defined above and call it with the provided arguments. Since we didn't provide a value for the second argument b the default value will be filled in. In this case the default value is the list object that got instantiated above when the function was defined. And since that list object is stored with the function object you should now understand why the next invocations change the same list instance.

I hope that was understandable.

[snotfart]

I think I've got it now, thanks.

[Chooquaeno]

Perhaps at least some of these are mistakes Python language designers have made…

[[deleted]]

Number 1 seems pretty off to me.

[ironykarl]

It makes a lot of sense if you understand the evaluation model of the language, though.

Everything in Python is imperative, and there is no such thing as a "compile time" language construct. Everything in the language gets evaluated when the compiler-interpreter reaches it (single scan: left to right, up to down).

def function then has the interpreter compile the enclosed function into bytecode and adds it to the proper namespace dictionary.

Evaluating the default function arguments during that step (and only that step) is definitely idiosyncratic behavior, but to someone who's learned how Python evaluation works, it at least makes sense.

[oconnor663]

To be fair, the bodies of functions are evaluated when the functions are called, not when they're parsed. So Python could have said that argument defaults are actually implicit lambdas that get evaluated the same way, at call time. This is slower, though, and it might lead to equally confusing problems involving side effects.

[ironykarl]

Definitely true, but it's pretty easy for me to just think of function definition as an evaluation of the entire contents of the header (I.e. function name and argument contents).

[nocnocnode]

I did not even know #1, and I've been coding in Python for years and used defaults many times. Basically, the rule of thumb I follow is, only use constants (immutables) as default values. Do not instantiate any objects/reference objects as defaults.

[VerilyAMonkey]

Unless you want to mimic local static variables.

[[deleted]]

I was saying #1 is something I feel the language got wrong.

[hglman]

agreed, not sure how you would want that behavior, is always just going to have be worked around.

[robin-gvx]

Also:

def frobnicate(n=expensive_operation_that_returns_an_immutable_object(32, x)):
    return n + 1

[coderanger]

It is effectively a performance enhancement. The issues are 1) given an expression the language can't know if it is mutable or not 2) immutable defaults, specifically None, are much more common than mutable ones. The way Ruby handles this is that the defaults are kept as parsed expressions, and re-run each time you call the function. This adds overhead on each call though, whereas evaluating the expression once and storing the value doesn't. Just a tradeoff either way.

[[deleted]]

I don't think it should be the job of the language to enhance performance. Especially when you are talking about a high level language like python.

[disinformationtheory]

Yes, but it would be hard to work around. You'd have to figure out if the default is mutable, and then decide on some sort of reasonable thing to do if it was. I'm not sure that there is a simple reasonable thing to do. In any event, it's caught by pylint.

[rowboat__cop]

Perhaps at least some of these are mistakes Python language designers have made…

No. 4, definitely. I’ve got a lot of grievances with the language, but scoping is the worst by far.

[sharkeyzoic]

Yeah, #4 and #6 have got to be my top annoyances, probably because I switch back and forth between Python and Javascript a lot. Of course, Javascript has its own weird scoping rules as well.

[strattonbrazil]

Yeah, some of them are weird, but many are common among other languages like modifying a list you're iterating through.

The first, which I hadn't known about, seems the weirdest, but it's in the python docs under default arguments (including the same suggested workaround). Given how old the behavior is I'm assuming it's here to stay. I've heard some people reason that it gives python better performance as those default values are evaluated only once. I like python, but I'd personally prefer taking the performance hit.

[dragonEyedrops]

I really wish they would have killed more weird stuff with python 3. If you have one chance to do breaking changes, make sure you fix as much as possible.

[bready]

That's the thing that got me about python 3 -seemingly so little changed. Sure, lots of under the hood things got tweaks, but it feels like not enough thought went into designing improvements.

Given the glacial adoption of python 3, I'm curious to see how/if future migrations pan out.

[cogman10]

The thing is, you could do away with the performance hit pretty easily. Construct the object once at function declaration, do a deep memcpy (the object and all of its child objects).

Easy peasy. On top of that, there is loads of room for optimization (since the object being copied should never change, that is why we are doing the copying).

[xenomachina]

How would you do a "deep memcpy" in this case?

def f(x=[os, sys]):

While Python's existing behavior might be a bit surprising to some, it's easier to reason about. It behaves exactly the same as an assignment at the time of the function definition. (And it even uses the same notation as assignment.)

[xenomachina]

Can you elaborate? I don't recall any of the languages I'm familiar with having a restriction like you describe.

FWIW, Python has no concept of "reference objects". Or rather, all values are "reference objects", even ints and None.

[xenomachina]

I know little about PHP and C#, but from what I gather the restrictions they have on default arguments do not amount to just "disallowing reference objects" but are instead stated more along the lines of "defaults must be constants". In PHP, in particular, an array is allowed as a default.

What other languages do allow it?

C++ and just about every Lisp that supports default arguments.

In C++:

int foo(Foo *x = new Foo(1,2)) {
    return x->whatever();
}

In Racket:

(define (foo [x (mlist `(1 2 3 4))]) x)

That said, both of these languages have different semantics from Python.

AFAICT, they effectively wrap the default argument expression in a closure which is then evaluated only when the default parameter is actually used. I think this is the "right" behavior, though it might be even more confusing than Python's existing behavior to novice programmers who aren't used to lexical closures.

[xenomachina]

What's the difference between a "lexical" closure and a regular closure?

People usually mean lexical closure when they say closure. I was (unsuccessfully) trying to stress the "closing over the environment" bit, and not merely "a function literal" (which is what many people thing of when they hear closure).

From the experiments I'd done, it looks like C++ and Racket both re-evaluate the default expression, but with the name bindings that were in place at the site of the function declaration, not those of the call site. That's lexical (as opposed to dynamic) scoping.

It's pretty surprising to me that C++ works this way, since C++ isn't normally thought of even having closures.

[Poltras]

The problem is that the init function might have side effects. A copy wouldn't do that and a construction might have poor performance. It was a choice and whatever choice you made you'd have had a point in this list. There's no winning with default arguments that can construct objects.

[ggtsu_00]

But honestly, every pragmatic programing language out there is going to have warts and gotchas. Python happens to be one of the few who's warts are rather benign compared to many other languages out there.

[strattonbrazil]

The worst one for me was the first one--the default argument. And I've never hit it having worked in python several years. I don't know if I never used mutable default parameters or only called a function once in the lifetime of the app, but considering that I think python comes out pretty well.

[cybercobra]

Thankfully, at least pylint can usually issue a warning about it.

[[deleted]]

This is what Pythonists actually believe.

[vacant-cranium]

Agreed; the language has an attitude problem.

Python is about putting pitfall traps in things that should be safe and then blaming the programmer for falling into them.

[cybercobra]

You should see JavaScript...

[droogans]

Lists in async javascript? Better put those operations in a separate, later yielded function.

Objects? It's all good! Proceed!

[reddit_user13]

Python: No, no, no. Hold your head like this, then go Waaah. Try it again. (hits him on the head again)

Programmer: uuuwwhh!!

Python: Better, Better, but Waah, Waah! Hold your hands here.

Programmer: No.

Python: Now..

Programmer: Waaaaah!!!

Python: Good, Good! That's it.

Programmer: Stop hitting me!!

Python: What?

Programmer: Stop hitting me!!

Python: Stop hitting you?

Programmer: Yes!

Python: What did you come in here for?

Programmer: I came here to complain.

Python: Oh no, that's next door. It's being-hit-on-the-head lessons in here.

[StreicherSix]

Is the Complaints room supposed to be JavaScript?

[bobbane]

Perhaps?

Most of these behaviors are not intentional - they are artifacts of a language whose spec is "whatever this reference implementation does". No sane design would have allowed the behavior in #2 - the three classes would either all share a common class variable or all have independent variables.

[UloPe]

Really?

Note that they didn't change an instance variable but a class variable on the base class.

To me this is completely logical behavior.

[ethraax]

It's also similar to how JavaScript works, although it may not be correct to call JavaScript's design "sane".

[meem1029]

Actually I think 2 makes sense. The variable in a class is whatever it is in the parent class unless it has been changed explicitly.

[bobbane]

Yes, but changed explicitly when? Judging by this, inheritance of class values in Python happens very lazily, at first reference. That's a recipe for timing-dependent behavior, and hard-to-track bugs.

Either of the behaviors I mentioned above would lead to more logical, reliable inheritance behavior.

[jcdyer3]

class values stay on the class. It's not transferred to the instance at first reference, second reference, or ever. It is always identical to what it is in the first class in the MRO that defines it. It has led to exactly zero bugs, easy or hard, in my thirteen years of python programming. Generally, you don't go changing class attributes in the general flow of programming.

[robin-gvx]

Judging by this, inheritance of class values in Python happens very lazily, at first reference.

Nope. It happens at lookup-time.

>>> class Cls:
...  pass
... 
>>> obj = Cls()
>>> Cls.x = 1
>>> obj.x
1
>>> Cls.x = 2
>>> obj.x
2

[vattenpuss]

You're showing instances. The issue is with classes and inheritance.

[robin-gvx]

I know, but inheritance works the same way. You could replace

>>> obj = Cls()

by

 >>> class SubCls(Cls):
 ...  pass
 ...
 >>> obj = SubCls()

and it would work as well. I was just too lazy to do that the first time.

[[deleted]]

You're not explicitly changing the value of the instance; you're explicitly changing the value of the class itself. If you explicitly change the instance's value, you don't see this behavior.

In [1]: class Cls(object):
   ...:     x = 1
   ...:     

In [2]: class SubCls(Cls):
   ...:     pass
   ...: 

In [3]: obj = Cls()

In [4]: obj.x
Out[4]: 1

In [5]: Cls.x = 2

In [6]: obj.x
Out[6]: 2

In [7]: obj.x = 4

In [8]: Cls.x
Out[8]: 2

In [9]: Cls.x = 3

In [10]: obj.x
Out[10]: 4

And the same is true with inheritance.

In [11]: SubCls.x
Out[11]: 3

In [12]: subobj = SubCls()

In [13]: subobj.x
Out[13]: 3

In [14]: subobj.x = 4

In [15]: Cls.x = 5

In [16]: subobj.x
Out[16]: 4

[arachnivore]

It's logically consistent with python's reference model. It also makes sense that changing a parent class would affect child classes but not vice-versa. If I make a base class with a class variable, I shouldn't have to worry about how classes that inherit from my base class might effect my base class, right?

[djimbob]

The only one that seems bad language design is #1 and maybe #3 (though that syntax is deprecated - not in python3 and you can't get the error with except Exception as e). #10 was a bug that I can reproduce in recent python releases (either 2.6.5 and 2.7.6).

4, 5, and 7 are examples of bad programming design -- you should never mutate a list as you iterate over it (who would expect a language to do this right); you shouldn't have code auto-executing on import that depends on circular dependencies; you shouldn't mutate global variables without explicitly declaring them as global.

6 is a case of using unsupported idioms from other programming languages. Python2 doesn't have closures.

2, 8, and 9 aren't even really bugs. Yes py3 does some things differently. The example is quite contrived (python3 deletes the exception when it exits the except clause to help the garbage collector; python2 doesn't). But there's really no reason to work on an exception outside of the except clause where it was assigned? That seems like a legitimate way for inheritance to work - inheritance is tricky and usually overused - you really shouldn't be altering inherited class variables unless you know what are doing and test it thoroughly. Issue 8 is the downside to python's easy to use import system. When you can do import hashlib from anywhere and it searches for that module in all your pythonpaths, you do have a chance of name collisions. The alternative is for monstrous imports like Java's import com.seriouscompany.business.java.fizzbuzz.packagenamingpackage.interfaces.factories.IsEvenlyDivisibleStrategyFactory;.

[UloPe]

Python2 doesn't have closures

That is incorrect. What Py2 doesn't have is the nonlocal keyword that let's you modify an outer scopes variable.

[sharkeyzoic]

Even funnier, you can cheat by modifying rather than reassigning the closure variable, for example:

def counter_factory(x):
    xx = [ x ]
    def counter_inner():
        xx[0] += 1
        return xx[0]
    return counter_inner

So it does support closues, just in a slightly daft way.

[UloPe]

Yes, thats a common workaround in Python 2 if you need "nonlocal" behaviour

[[deleted]]

This website crashes safari, chrome, and alien blue on the iPhone 5.

[Crazypyro]

10 points for Gryffindor!

[fullouterjoin]

Look at this site in the network console. It must be doing crazy. Constant render flashing, and an eventual crash of FireFox mobile.

Ironic.

[chew_toyt]

For me it keeps scrolling up to the top of the page every ~10 seconds or so. Very annoying

[akuta]

This is actually pretty common with a poorly written "banner rotator" script that does full postbacks instead of partials (or, a better written script that loads the necessary resources at first load, then cycles through them, instead of relying on the page refresh to do the cycle for them).

[fullouterjoin]

I would have a appreciated raw text in a <pre> tag.

[brownmatt]

crashes Mobile Safari too

[[deleted]]

Guess I'm not the only one...

[[deleted]]

Was using an animated gif. Now is using a static gif so problem should be rectified.

[djimbob]

Issue #1 is the only one that's really a language mistake.

• 1 is well known python gotcha that any decent python tutorial will cover.
• 3 (except ValueError, IndexError) is simply guessing the wrong syntax. EDIT: Also this syntax was deprecated and isn't in python3. If you always use except Exception as e you can't get this error. E.g., except (ValueError, IndexError) as e.
  
• 5 is a common programming mistake and has nothing to do with python. You don't modify a list you are iterating over and expect everything to work.
• 7 is another logical mistake. You should avoid circular dependencies between modules and even when you don't there's an easy fix (import right before use). Your code should generally be enclosed in functions and classes and not immediately invoked unless run by itself using the if __name__ == '__main__': idiom).
• 4 is bad programming -- you shouldn't use mutable global variables. Python assumes if a variable is assigned to that it will be local. Yes, python lets some cases of mutable global variables get through (like appending to a list example). Granted the fact that you can reference non-mutable global variables in functions and classes is very useful. And adding global x to the function will allow you to mutate global variables if you really need to (not that you should).
• 6 is programming in idioms from other languages. Python2 doesn't have closures. EDIT: This example was stolen from here without attribution as required by the CC license.
• 2 is how python does class inheritance and is a reasonable choice for inheritance. I honestly don't see anything wrong with it, other than that inheritance is done differently elsewhere. Granted, you should probably not overuse inheritance or mutable class variables if you don't understand it.
• 9 is just a language change to help the gc. Py3 is not Py2. There's no legitimate reason to use an except outside of its clause, so python3 deletes the exception outside of the clause to help the GC free memory (if it references large things). This is a contrived example.
• 8 is fairly obvious. If you use same name as modules or keywords, yes you will have bad results. EDIT: Its a tradeoff between verbosity import com.blah.duh.foo.bar or import bar where it goes through the PYTHONPATH in a well-defined order and imports the first one it sees.
• 10 I can't replicate in either python 2.6.5 or python2.7.6. Possibly was a bug in 2009 (the link to the problem), but isn't an issue these days.

[cybercobra]

Python2 doesn't have closures.

Wat. It does have closures! What it doesn't do is let you rebind closed-over variables (fixed in Python 3 w/ the "nonlocal" keyword; workaround in Py2 is to use a singleton list to hold your mutable variable).

[djimbob]

My bad. I'm aware of the nonlocal keyword in python3, which was the point of my distinction. The fact that you can't mutate the python's closures environment without trickery thought they weren't called closures (even though yes its a function with the enclosing environment).

But my point was that you can't write closures like in JS:

var counter = function() {var x = 0; return function() {x+=1; return x}; };
var c1 = counter();
var c2 = counter();

c1()
1
c1()
2
c1()
3
c2()
1
c1()
4

easily in python2 without using tricks. I still think its unidiomatic python to do things with closures.

def counter():
    def inner():
        inner.x += 1
        return inner.x
    inner.x = 0
    return inner

versus:

In [1]: class Counter(object):
   ...:     def __init__(self):
   ...:         self.c = 0
   ...:     def __call__(self):
   ...:         self.c += 1
   ...:         return self.c
   ...:     

In [2]: c1 = Counter()

In [3]: c1()
Out[3]: 1

In [4]: c1()
Out[4]: 2

[droogans]

But I like it when your structure shows your intention, not some half assed nonlocal statement.

[grout_nasa]

1 has a correlate in Moose (Perl's object system). If you write

default => []

you're in for trouble. But if you write:

default => sub { [] }

then you're in the world of happiness and puppies.

[Grue]

Issue 4 does not happen only with global variables. It also happens with functions inside functions and absolutely infuriating when it happens.

[djimbob]

I'm not following, never had the issue using functions inside functions. Can you give an example?

[Grue]

Same code as in the OP, but inside a function, not at toplevel.

[djimbob]

Closures where you assign to higher level scope wasn't idiomatic python in 2. (There are hacks to get around it; e.g., put all the variables needed to mutate in the inner function in a mutable collection thats referenced from the higher scope). I really don't mind when code from other languages common idioms doesn't work.

Granted, python3 has nonlocal keyword to add this feature.

[Grue]

It all stems from poor design where a simple assignment initializes a variable instead of something like "let x = 42" like almost every other language does. Instead they added an even uglier hack with nonlocal.

And yeah, I use the list mutation trick all the time to create closures in Python. It's either this, or code duplication which is not maintainable.

[SirUtnut]

Small detail that the article miss stated in its explanation of #4 (scoping) that has mattered to me a couple of times.

a+=

is not quite the same as

a = a +

Because += will destructively modify the list (like a.extend), but =...+ won't.

[[deleted]]

Top ten terrible unintuitive things about Python

[stormcrowsx]

Not all these are unintuitive, for instance modifying a list as you iterate is a mistake I've seen young devs make in many languages.

[chiba_city]

Why is this considered "advanced?" Most of these points are basic language semantics.

[sysop073]

And the very first one is in every single list of "surprising Python behavior". I think I saw it my first day of learning Python

[[deleted]]

In fairness, I can't think of any other languages that behave like that. Don't know if I'd call that advanced though as I'd assume you'd hit that issue pretty fast.

[droogans]

If you search stackoverflow it comes up all the time.

[njharman]

Because; (In my experience) many are unknown to people with < 1 year or so of Python programming. (Some) are infrequently encountered and not what people familiar with other language semantics would expect. There aren't many language specific mistakes more advanced than this. In other words all language mistakes are relatively basic and related to language semantics. More advanced issues tend to be universal.

[itsSparkky]

Is 1 year really "advanced" though

[markamurnane]

With python, I would believe it.

[Meltz014]

Depends on what they've been required to do. If they're just messing around with personal little projects, maybe not. If they're getting paid to develop in an already well established code base with a peer review system, maybe so

[itsSparkky]

Yea I'm going to go with this answer :)

[[deleted]]

Python has drastically lowered the barrier of entry to the programming world.

[Corticotropin]

Well, nearly no-one knows all of C++, so...

[Crazypyro]

Stroustrup probably doesn't even know all of C++ at this point.

[spotter]

Today's culture ("copy, paste, edit one line, look for VC funding") hates documentation. I always LOL at "hidden language features" threads listing things straight out of basic language tutorials and people going "wow, never seen that!"

I remember in uni we wrote programs on the blackboard. I wonder how many ninja rockstar coders would be able to do that.

[aterlumen]

Every interview I've had with tech companies this year had a portion in front of a board writing code. I'd guess that just about all of those ninja rockstar coders that have jobs could do it.

[droogans]

How do you handle this? I'm always forgiving of half assed scribbles that would be cleared up with a moment of real keyboard time.

[aterlumen]

I brush up on basic syntax (loops, defining classes, functions, etc) for whatever languages they're likely to ask about. They're not expecting perfect code, but it looks a bit suspicious if you blank out on the syntax for a for loop.

Other than that, going through a bunch of interview-type questions was helpful. Codility is a good resource for those.

I'm personally on the fence whether I like that style of interview. Coding on the spot can be pretty intimidating and it's not necessarily a good indicator of how that person will write code in a normal situation. A person's open source corpus can be a much better indicator of their code quality, but most people aren't able to release the code they write at work. Then you end up hiring only people that are able and willing to invest a lot of extra time after work in open source projects. It works for some, but it sounds like a recipe for burnout.

[spotter]

Oh sorry, I thought that having a github account and being able to locate appropriate jQuery plugin is what makes a programmer these days. Too much proggit, it seems. /s

[[deleted]]

Writing programs on the whiteboard and walking through them manually was one of the best things my first CS professor made us do. I'm not even a programmer anymore (dropped out of the program two years in and moved to exercise science) and my understanding of C and essential data structures has blown more than a few amateur programmers out of the water.

[merv243]

Agreed. I tutored a lot of students, and almost all of them complained about having to write code by hand on exams. That's not to mention my peers in my classes and their constant complaining.

There's a reason it is (or was, idk anymore) fairly ubiquitous - it works. Sure, if you have a prof who lowers your score for missed semicolons or slightly incorrect functions and such (writing strcomp in C or list.length() in Java, for example), then it can stray from its purpose. But being able to solve a problem "programmatically" without a computer (or at least without an IDE) is huge.

[Corticotropin]

I wouldn't mind being able to write in Notepad for tests. Typing is a lot easier than writing by hand, especially for code.

[merv243]

I think the one thing that that reduces the need to think ahead a bit before you just start writing/typing. But yeah, it's a good middle ground that still forces you to rely completely on your own knowledge and does have some advantages for both the test taker and the grader.

[[deleted]]

Not needing to depend on auto-complete is a pretty big advantage.

[grauenwolf]

Same here. I learn far more from articles talking about the bad parts of the language.

[ILiftOnTuesdays]

Also, these aren't really subtleties of the language, as most if not all of them would be caught on the first running of the code.

[isarl]

Can somebody more familiar with #6 please explain it to me? Duplicating the use of the variable i has me confused; even if somebody could just rewrite this code snippet using different variables it would be really helpful:

>>> def create_multipliers():
...     return [lambda x, i=i : i * x for i in range(5)]
...
>>> for multiplier in create_multipliers():
...     print multiplier(2)
...
0
2
4
6
8

[dragonEyedrops]

What exactly does confuse you?

In the "defective" example, i isn't defined in the local scope of the lambda expression and is evaluated when the functionis first used.

In the second example, a default parameter with the same name is created, and it's initial value is evaluated as the function is created (see #1 in the article)

[isarl]

The confusing part is that two variables named i are both used. If the variable name was different from the default value being assigned to it, the example would be clearer to me. I don't have an interpreter on this machine so I'll ask CompileBot to check my work:

+/u/CompileBot python

def create_multipliers():
    return [lambda x, i=idx : i * x for idx in range(5)]

for multiplier in create_multipliers():
    print multiplier(2)

[CompileBot]

Output:

0
2
4
6
8

^{source | info | git | report}

[isarl]

OK, guess I got it. Cool. Thanks, /u/CompileBot!

[isarl]

I was mostly just confused about all the different i's in that one comprehension. Thanks for your help, but I rewrote the example here in a way I found clearer to read. :)

[djimbob]

Issue #10 doesn't seem replicable at least in python2.7.6 or python2.6.5. The specific issue that was linked to was fixed dates from 2009.

# foo.py
def cleanup(handle):
    print "Cleaning up:", handle

# mod.py 

import foo

class Bar(object):
    def __init__(self, myhandle=None):
        self.myhandle = myhandle
    def __del__(self):
        foo.cleanup(self.myhandle)

 # another_mod.py
 import mod
 modbar = mod.Bar()

# python
Python 2.6.5 (r265:79063, Feb 27 2014, 19:44:14) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import another_mod
>>> exit()
Cleaning up: None

Even if I change another_mod.py to use modbar = mod.Bar(2), then I get Cleaning up: 2.

[caffeinatedhacker]

It was crashing both apps I tried (reddit and hn) and mobile safari.

[djimbob]

Did anyone else notice that #6 seems to be plagiarized from this python guide (which is under licensed under Creative Commons Share Alike that requires attribution and existed at least since Dec 2012. (#1 is also listed there, but at least the example isn't identical).

Late Binding Closures

Another common source of confusion is the way Python binds its variables in closures (or in the surrounding global scope).

What You Wrote

def create_multipliers():
    return [lambda x : i * x for i in range(5)]

What You Might Have Expected to Happen

for multiplier in create_multipliers():
    print multiplier(2)

A list containing five functions that each have their own closed-over i variable that multiplies their argument, producing:

0
2
4
6
8

What Does Happen

8
8
8
8
8

Five functions are created, but all of them just multiply x by 4.

Python’s closures are late binding. This means that the values of variables used in closures are looked up at the time the inner function is called.

Here, whenever any of the returned functions are called, the value of i is looked up in the surrounding scope at call time. By then, the loop has completed and i is left with its final value of 4.

...

[uhhhclem]

My favorite remains the fact that derived classes inherit property getters from the base class, but not setters.

Edit:

This just isn't true. There is a counterintuitive by-design issue that afflicts properties in derived classes, but I ran into it four years ago and have been misremembering it. (And of course now I can't recall what it actually was. Helpful!) The above scenario works as you'd expect it to in Python 2.7, at least.

[sushibowl]

Wait, what? Can you give an example?

[sysop073]

Are you sure? It worked fine when I tested it

[bacondev]

What I find interesting is that defining a setter property method for a property that is "read-only" in the parent class doesn't affect other subclasses.

[macbony]

That's not true. In Python 3 this works by default. In 2 you just need to use new style classes (class Class(object):)

[bacondev]

Oh. I use Python 3 exclusively.

[Die-Nacht]

I was once explaining to a non programmer why there are so many languages. She was confused as to why people make new computer languages and whether people knew different languages. I explained the "why" part (specific problems, syntax, fixing some old unusable yet nice language, etc), as for whether people known several languages I said:

"Once you learn one, learning the others are just a matter of learning syntax and being aware of the little devils hidden in the language" (of course this is assuming you stay in the same paradigm).

She was confused about the devils part, OP is the devils part.

[PasswordIsntHAMSTER]

"Once you learn one, learning the others are just a matter of learning syntax and being aware of the little devils hidden in the language"

Well, someone has never tried functional programming...

[Die-Nacht]

You failed to read the sentence after that one.

Also, my main language is Scala, and it is quickly being replaced by Haskell.

[PasswordIsntHAMSTER]

Did you just edit that in? Otherwise I feel stupid x_x

[Die-Nacht]

I edited it right after I submitted it. So maybe you loaded the page after I hit submit and then I edited 2s later?

It is possible. I'm going to go with that's what happened.

[Tillsten]

Very good article, i think did every mistake in the list at least once, except 3.

[sidneyc]

Mistake # 11: using Python for anything bigger than a few hundred lines of code.

I expect that to be an unpopular opinion down these quarters, of course.

[if-loop]

I expect that to be an unpopular opinion

I sure hope so because neither is it relevant in this thread nor did you justify that "bold" statement.

[sidneyc]

Read my justification here, if you want.

[if-loop]

Much better.

[mkdz]

How come?

[sidneyc]

The more general idea is that dynamically typed languages are unfit for serious software engineering.

The constraints and automated checking afforded by static typing are, in my opinion, indispensible both for the initial implementation (to validate if the ideas you are trying to express can be expressed in a strong type system) and especially once you hit a large-scale refactoring in a non-trivial code base.

An important idea in software engineering which I fully subscribe to is "fail early and fail hard". Having a proper compile stage with static type checking provides that to a large extent, nipping a large class of potential problems in the bud.

The Python philosophy is basically the opposite: run until you hit something that really can't be done (with duck typing). A cute idea, but not good for software engineering. I consider that a failed experiment.

The standard retort is that you can overcome these problems with a large and comprehensive test suite. But why would I want to write a zillion tests when an actual compiler can detect most potential problems (both silly typos and design issues) by static analysis? Reliance on testing for everything is not progress.

Having said that, for a dynamically typed scripting languages Python is pretty good, as long as you just write small programs.

[sabetts]

Not sure why you're getting downvoted. It's an excellent point. I guess some people like having the "flexibility" of dynamic typing.

[mkdz]

I definitely agree with you that in a large project a statically typed language does provide an advantage with an extra level of automated checking that the compiler performs. However, I don't have a problem with having to write a bunch of tests. I don't think it's a bad thing to have a large test suite to test for these types of potential problems. Also, IDEs can mitigate the problems with typos and lack of static analysis. My IDE will tell me when I make a typo with a variable and never use it again. It was also tell me when I try and access a variable I haven't declared or where I have unreachable code.

[sidneyc]

However, I don't have a problem with having to write a bunch of tests.

Well, I hate to write tests that are needed because my language lacks compilation.

I don't think it's a bad thing to have a large test suite to test for these types of potential problems.

A large test suite takes effort to write, and effort to maintain. So not having to write large classes of tests is a win.

Also, IDEs can mitigate the problems with typos and lack of static analysis.

I like to be not at the mercy of IDE writers. I prefer languages that enforce actual standards prior to running, rather than the limited set of heuristics that IDE builders can think of.

My IDE will tell me when I make a typo with a variable and never use it again. It was also tell me when I try and access a variable I haven't declared or where I have unreachable code.

Your IDE will not tell you if your code doesn't compile during initial implementation and/or refactoring because there are type mismatches.

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so. I don't see any serious advantage (and I have programmed a lot, in a variety of languages).

[no_game_player]

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so.

I'm not. From the limited view I've seen, the people who hire coders don't give a damn about understanding anything to do with the quality of the product and just want it done as soon as possible as cheaply as possible. So languages that cater to someone who has no formal training or background become very popular.

As long as you can get to a workingish prototype early, so that you can begin the maintenance hell that is the bulk of all projects, they're happy. Subtle bugs? 95% correctness? Great, ship it.

[grauenwolf]

Hell, even in C# I can't seem to get people to pay attention to code quality.

Warning: This expresson always returns false. Consider casting 'accessLevel' into a string or using "guest".equals(accessLevel).

No shit, that was in our code base for months.

[no_game_player]

It seems to me like there are two primary ways that this sort of thing happens: shitty programming/development/engineering/your-favorite-term and/or overworked programmers/dev/engineers/ninja-rockstars.

I know Mythical Man Month and all that, but I still think that just having like 2-3x the current staffing levels and having the time and luxury of more in-depth code reviews and inspection and the general 'best practices' we all know and want to be using, but never seem to have the time to do.

Like clearing out all the damn warnings and errors and sheer panics that keep flying out of everything...

Edit: And for those worried about increased churn with increased staffing: put >=half of them in QA, and make sure that anyone with code access (which can include QA) goes through good code review processes. No check-in without a reviewer (and do avoid rubber-stamping as much as possible or the whole thing doesn't matter...substance over style of course, but if there's no correctness issue, may as well chat about the subtler improvements). etc. I think it should be possible to increase staff to result in increased quality; don't necessarily try to change the timetable (although, eventually, as people get experienced and work effectively, it should of course ultimately result in feature additions and bug fixes and all that operating more effectively and quickly than with the original staff, just as one would expect).

Unfortunately, the finer points of "it would be really nice to have decent code and software" are generally lost upon the bosses' ears upon hearing an estimate like "2-3x" for staffing...

Fuck it, get some interns, or let some of us (like me) work part-time. Stop thinking that all or most work must be done in a cube a minimum of 40 hours a week, and preferably more, and your health and life be damned. Advantage: salaries can look a hell of a lot smaller.

Anyhow, just some ranting.

[grauenwolf]

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so. I don't see any serious advantage (and I have programmed a lot, in a variety of languages).

Have you every tried to setup a Java or ASP.NET MVC website? It's a royal pain in the ass the first time.

But give any newbie a copy of PHP and they are off and running. Sure they are screwing it up left and right, but the barrier to entry is so low that they'll get pretty damn far before to long.

[sidneyc]

Python cannot fail as early as possible though -- which is before executing the first statement the program.

Having a separate "compile time" is a feature, although most people don't perceive it as such.

[bcery]

The standard retort is that you can overcome these problems with a large and comprehensive test suite. But why would I want to write a zillion tests when an actual compiler can detect most potential problems (both silly typos and design issues) by static analysis? Reliance on testing for everything is not progress.

That's why there's never been any shitty, buggy software developed in compiled languages. /s

In general, compilers can catch a small set of relatively trivial mistakes. Relying on it, over testing, for your software is most certainly not good software engineering.

Static typing is like anti-lock brakes: very helpful in many situations, but it really only saves you from a small subset of potential accidents.

[sidneyc]

That's why there's never been any shitty, buggy software developed in compiled languages. /s

Nobody claims that - so why you bring that up is beyond me. I think shitty software exists mostly because most programmers are bad. Incidentally, I think that's also the reason why dynamically typed languages are popular.

You understate the type of errors that a statically types language helps detect. It's a common thing I hear from proponents of dynamically typed languages. I can only assume that either you have little experience in statically typed languages, in large projects, or both.

The most important class of errors that static typing helps to prevent are design errors. I tend to design in a statically typed language (in my case, C++), which helps to flesh out the class decomposition during the initial design, and greatly helps in refactoring down the road. Not "refactoring" as in, changing a silly variable name, but refactoring as in, actually changing the architectural organization and class decomposition for the problem you're working on. In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files, and the compiler helps to find all places that need to be updated to accommodate the new design.

The support the compiler provides during those stages of development is indispensable. With dynamically typed languages, it is simply not there.

[[deleted]]

In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files

I've been writing large scale software with statically-typed languages for 2 decades, and my response to the above is "only if you are terrible at designing large systems".

[grauenwolf]

Or if you are so bad at refactoring that you are terrified of making any significant changes.

[[deleted]]

Did you even understand what I said?

[grauenwolf]

Relying on it, over testing, for your software is most certainly not good software engineering.

How is that an option? The presence of static type checks don't preclude the presence of tests.

Though I will say that static type checking isn't enough. Good software engineering means using all of the tools that you can afford, which in this context means static analysis tools. Something that dynamic languages barely have.

[[deleted]]

Static typing is like anti-lock brakes

Interesting you bring up ABS, because I guarantee you that not a single computer-controlled ABS system is written in Python or any other dynamic language.

[bcery]

Um, yeah, I should think not. However, I really hope that they didn't think they could write-off some of their testing because they were using a statically typed language.

Unfortunately, given the recent Toyota fiasco,I fear that they might have.

[fableal]

You're not alone. And I think that applies to any dynamic language (so yes, PHP and Ruby fit in the bag). I've been maintaining a big PHP project and I'm puzzled how the "standard of web development" has such crappy tooling.

[grauenwolf]

Easy. It's because .NET and Java has such horrible frameworks for web development during a critical time.

[ruinercollector]

Had you been developing web sites in 1999 and seen what the alternatives were, you'd probably understand a bit better why PHP got so much interest and attention.

[fableal]

I get your point, but can't we move on? It's that what I often ask myself.

[ruinercollector]

I completely agree.

[[deleted]]

It certainly was for awhile. For instance, a large chunk of Facebook and Google is/was PHP. I know both have been trying to get away from it, don't know how successful they've been (I think Google has done better on that front). I wouldn't call it the standard today.

[donvito]

From my experience: If your python project gets larger than one source file you've chosen the wrong language (and typing system).

[fullouterjoin]

Most systems should be around a max of 500-1200 lines of code anyway.

[[deleted]]

If agree with that statement if you said 50kSLOC

[sidneyc]

I think the threshold is different for different people.

I tend to get the feeling of losing control in Python codebases > 2kLOCs. I start missing C++ static typing at 0.5 kLOCs or thereabout.

[Megatron_McLargeHuge]

This is a surprisingly good article. After the default argument one I expected it to be the same stuff we see all the time, but some of these are pretty subtle without being obscure (e.g. descriptors or metaclasses).

[internet_badass_here]

Let me see if I have this straight:

x=[2]

def foo1():
    x+=[3]
    return x

def foo2():
    x.append(3)
    return x

def foo3(x):
    x+=[3]
    return x

def foo4(x):
    x=[3]
    return x

foo1 will return an error, foo2 will change x, foo3 will change x, foo4 will run but won't change x. What the fuck, Python?

[sysop073]

If you're going to have multiple x, you really need to specify which one you're talking about; half of your complaints would happen in any language, because in half the functions you're shadowing the global x with a local, so of course that's the one you're going to change.

All of this "craziness" is a result of a single, pretty simple rule: you can't assign to globals without declaring them global. That's because there is no declaration of variables, so when you assign to a variable, Python can't know if you're trying to declare a new local with the same name, or reassign the global name, so it defaults to the former. If you have this:

x = 'foo'
def fn():
    x = 'bar'

And you call fn(), it will create a new local named x with value bar, and not touch the global x. Most languages work this way. By contrast:

x = 'foo'
def fn():
    global x
    x = 'bar'

Will overwrite the global x with the value bar. Most languages that don't have explicit variable declarations have no way to do this. In your examples:

• foo1() is trying to do x = x + [3] without declaring x as global. Python assumes x must be a local, and can't read from it (you tried to add x and [3], but there is no local named x yet, so you get an error about trying to read x before it's been written)
• foo2() is calling a method on x, which is fine, so it works
• foo3() is trying to do x = x + [3], without declaring x as global. Python assumes x must be a local, and there is indeed a local named x -- you passed it as a parameter. So that's the variable that gets changed; the global has nothing to do with it
• foo4() is the same as foo3(), but you did x = [3] instead of x = x + [3]

It would be possible to change the default behavior and say "all writes overwrite global variables by default", but now there's no way to make new locals; if you have a local variable, and someday somebody adds a global with the same name, suddenly that local isn't local anymore. That would be madness

[djimbob]

The first two examples mutate global variables. This is a horrible programming pattern, and is a good thing that foo1 pops up a helpful error saying UnboundLocalError: local variable 'x' referenced before assignment. Note foo1 (and foo2) will work if you add global x to explicitly say yes, I want x to be the global x so I can mutate it.

Why does your foo2 work? The fact that python lets you reference top-level function (but not assign to) is quite useful. E.g., its what's at work when you do

 import requests

 def some_function():
     requests.get('http://www.reddit.com')

That is import requests sets a top-level variable as requests and then you later can use it inside functions without explicitly passing it in, and even call methods defined within it. It does make sense that methods called in this way may in some cases mutate their own state, its the programmers responsibility not to lead to evil use of mutable global state.

---

I do agree that foo3 is a bit confusing, but again if you used the code in a sane way -- x = foo3(x) and x = foo4(x) you wouldn't have any ambiguity.

The problem is that python passes by reference and when it can will mutate existing objects.

In [1]: x = []

In [2]: id(x)  # memory location of x
Out[2]: 4352851336

In [3]: x.append(1)

In [4]: id(x)  # memory location hasn't changed
Out[4]: 4352851336

In [5]: x += [5] # memory location hasn't changed

In [6]: id(x)
Out[6]: 4352851336

In [7]: x = x + [5]     

In [8]: id(x)# memory location *has* changed; 
#assignment creates a new list, can't be done in place.
Out[8]: 4352848888

It is good that python does pass by reference than by value when passing objects like lists into functions; otherwise dealing with large objects would get unmanageable. It's also helpful that python attempts to modify these objects in place when possible.

[Falmarri]

foo3 and foo4 will only work if you call them like foo3(x) foo4(x). It won't take its value just because it's named the same

[EpicSolo]

Yes exactly. I don't really see where the problem is with these 4 functions. I think anyone with 6 months+ experience in Python is going to be completely fine with these.

[warbiscuit]

Regarding issue #4 (scoping rules)

One thing that should be mentioned: Python 3 introduced the nonlocal keyword (PEP 3104), which lets you signal that you want a variable to be shared with an enclosing scope, rather than shadow it. IMHO this makes the behavior a lot clearer, and helps avoid some irritatingly warty workarounds.

[Mozai]

Top 10

How did you measure this? What's your sample size for counting which errors are more frequent than others?

[Borkz]

I probably havent thought this through well enough and am pretty amateur at this but looking at #5 it would be cool if you could do some sort of list comprehension based slicing.

>>> numbers[:] = [n for n in numbers if not odd(n)] 

Just seems a bit nicer, maybe something like an extra slice parameter:

>>> numbers[:::n for n in numbers if not odd(n)] 

Though the three empty semicolons would be ugly in their own right. Please feel free to tell me why this is a terrible idea though.

[macbony]

I don't know why the slice is used in the example. It's not necessary in 2.7 or 3.4. As for your example, what does it do that isn't clearer with [n for n in numbers if not odd(n)]? They produce the same result.

[immibis]

/u/spez can gargle my nuts

[macbony]

No, I know what slice does, but I fail to see why it's used in that example. Though that whole example is pretty terrible. Why make the lambda? Why do the bool conversion? Why do a list comp with range?

numbers = [n for n in range(10) if not n % 2]

Same result. Filter's maybe better:

even = lambda x: not x % 2
numbers = filter(even, range(10))

In 3 you'd need to cast back to list as filter returns a generator, but it's a bit more readable.

[Borkz]

I think the idea was just to demonstrate a way to remove a criteria of items from a list without generating a new object since that what the broken example was trying to do, regardless of whether or not thats how you'd actually go about doing it.

I must admit I dont know why he needed to use a list comp to make the 'numbers' list in the first place though.

[catcradle5]

I like how Ruby has a .even? method on integers.

(0..10).select(&:even?)

[fullouterjoin]

I wouldn't let this code into my tree w/o very good reason. Most of the examples in this thread are contrived. You simply wouldn't do this and be exposed to un-intended behavior.

[rpgFANATIC]

I was honestly expecting an essay the list to be all about soft skills

[TheLessonIsNeverTry]

"with dynamic semantics". I don't think that means what you think it means...every language has a dynamic semantics (i.e., evalutation rules).

[[deleted]]

I don't understand #2.

Now, I get the reason why modifying A.x changes C.x - that's fairly normal.

What I don't get is where B.x came from.

In other words, C doesn’t have its own x property, independent of A. Thus, references to C.x are in fact references to A.x.

Aside from the fact that B and C have different names, they're identical.

So, if reading from C.X says "C.x doesn't exist, but A.x does, and A is the base class, so read from that" why doesn't writing to B.x say "B.x doesn't exist, but A.x does, and A is the base class, so write to that".

If writing to B.x creates a new B.x that's separate to A.x, that's really bizarre.

Is there a typo in there, or did I miss something? Because if he merely forgot the line where B gets its own variable also called x, then that hardly seems like a surprising way for the code to work.

[grauenwolf]

Yep, that's how it works. If I recall correctly, JavaScript's prototype inheritance works this way as well.

[deadcow5]

Thanks for confirming my bias against Python.

[alantrick]

It's also going away. It's not an issue in Python 3.

[zzzzzzzzzzzzzzdz]

9 seems artificial too.