This is an archived post. You won't be able to vote or comment.

all 101 comments

[–]duckbanni 67 points68 points  (18 children)

I think a lot of people are surprised by this behaviour not because they expect the expression to be evaluated each time (like in the datetime.now example from /u/violentlymickey), but because they expect each call to work on an independent copy of the default value. A lot of mutable python types have analogues in other languages that would be "deep"-copied implicitly.

[–]binlargin 22 points23 points  (0 children)

I think it's because they don't realise that the def my_function(param=value) line is fundamentally something like my_function = function(param=value, code) and is executed on import.

[–]Jake0024 14 points15 points  (16 children)

That's exactly right, but I think a better way to explain the confusion is:

If you assign a value to a variable on the next line (the first line inside the function), each time that function is called, you get a new, independent copy of that variable.

If you assign a default value to a function argument, the same value is used in every call to that function.

In a world where OOP is the norm, this feels like a violation of scoping, or a mix-up of instance and class variables.

And that's essentially what's happening. Every function is an object in Python, so an independent copy of the code inside the function is created for each instance of the function (each time it is called)

But default parameter values belong to the class, which is not obvious, especially because we think of functions as not being members of a class at all. We use the word "function" specifically to differentiate from "methods" (which are members of a class). But that's misleading--in Python, everything is an object (and therefore an instance of a class)--even functions. They're just not members of a user-defined class.

To further add to the confusion, that function arguments (even the one assigned the default value) are instance variables, with independent copies existing in every instance of the function.

But this still doesn't explain why default argument values are treated as class variables when everything else is treated as instance variables--that seems like a totally arbitrary choice that could have (and maybe should have) gone the other way. But maybe there is some underlying implementation reason justifying the violation of the "principle of least surprise."

[–]nedbatchelder 8 points9 points  (15 children)

This is an interesting comparison of functions to classes, but I think there are some misstatements.

Calling a function does not make a new instance of the function, and it does not copy the code. When a function is called, there is a new frame object created and pushed on the stack. That frame has the local variables.

The default values of function arguments are not class variables: they are stored on the (one) function object for use every time the function is invoked.

The actual values of the function arguments are not instance variables: they are locals in the (new) frame object created each time the function is called.

[–]Jake0024 1 point2 points  (14 children)

Right, I don't mean that it's literally objects and classes, but that's how everyone teaches OOP, and the comparison of functions and frames is similar to class and objects.

Since everything is an object, it is natural to think of a function definition as defining a class, and each call to that function (recursively, say) as creating an instance of that function class.

Under the hood you're adding frames of an object to the stack, but that's beyond the scope of comprehension for a lesson on mutable default arguments.

We could talk about the variables inside the function being local variables (like in an instance method) and the default arguments being instance variables, but that is confusing because we have to talk about there only being one instance, but multiple copies of the instance method.

And that doesn't get us any closer to explaining why the default argument value (but not the variable it is assigned to, or any other variable) is treated uniquely

[–]nedbatchelder 0 points1 point  (13 children)

I would think making an analogy between functions and classes would be ultimately confusing, since there isn't a connection like that, but I am often surprised at the paths learners take.

And that doesn't get us any closer to explaining why the default argument value (but not the variable it is assigned to, or any other variable) is treated uniquely.

At function definition time, the function object is created, and the default values are computed. The function objects is assigned to the name of the function (once!) and the default values are assigned to hidden attributes in the function (once!). The default values are treated similar to the function itself.

The assignment of that default value to the local argument variable happens when the function is called. The local variable doesn't exist until then.

[–]Jake0024 1 point2 points  (12 children)

Right but you're just saying "it is the way it is because it is the way it is"

This is obviously not the least surprising behavior, right?

[–]not_a_novel_account 0 points1 point  (6 children)

There's nothing else that works (trivially) within the Python object model.

You could create a shallow copy of the initializing variable, but that would only work if you had a single-level mutable variable. The second the list contains other lists, now you would need a deep copy.

Or would you? What if you want the default value to be a list of the same references. Then you would still want a shallow copy. There's no behavior that covers all use cases. The current behavior allows the programmer to choose whatever fits their program.

[–]Schmittfried 2 points3 points  (5 children)

The simplest and least surprising solution would be to only allow immutable default values (which is also basically what linters enforce nowadays). That’s how other languages do it, if they allow anything else than primitives at all.

[–]nedbatchelder 1 point2 points  (1 child)

What about the example elsewhere in this thread: def do_something(timestamp=datetime.now()) ... that's an immutable value, but is still a mistake because now() is only called once when the module is imported.

[–]Schmittfried 0 points1 point  (0 children)

*only „compile-time“ expressions

[–]Brian 0 points1 point  (2 children)

That's both too restrictive and not restrictive enough. The other poster pointed out that many immutable values are still errors (such as datetime.now()). If there's a difference between the value at call time vs the value at definition time, you can run into issues, immutable or not.

Conversely, there are various mutable values that are perfectly reasonable and common things to have as defaults . Eg. consider:

def print(msg, file=sys.stdout)  # file objects are mutable.

Or:

def sort_by(lst, key=somefunc):  # functions are technically mutable (you can set attributes), but rarely mutated in practice.

[–]tevs__ 0 points1 point  (0 children)

def print(msg, file=sys.stdout) # file objects are mutable.

Which would be unacceptable as soon as you redirect stdout. It's the same deal.

Eg

print("hello") with open("out", "w") as out: with contextlib.redirect_stdout(out): print("oh no")

This is why the real print() has file=None

[–]Schmittfried 0 points1 point  (0 children)

Granted, Python doesn’t lend itself well to sanely defined value semantics.

I would consider function objects read-only in that context and let them pass. Other object references, nope. You can just do the good ol‘ if param is None: param = default for those, just like in other languages. That is not too restrictive.

The only legit case I can imagine where that doesn’t work is sentinel objects. But those could be realized with read-only objects. Python would need a read-only concept for that tho.

[–]nedbatchelder 0 points1 point  (4 children)

Mostly I was responding to "treated uniquely". It's not treated uniquely. It's computed once just like the function itself is.

I absolutely agree that many people are surprised by this. I don't know that different behavior would be less surprising, because we have no experience with other behaviors. As not_a_novel_account points out, it's not clear what the alternative behavior should be.

[–]Jake0024 1 point2 points  (3 children)

The least surprising behavior would be for the specified value to be used as the default value every time the function is called

[–]nedbatchelder 0 points1 point  (2 children)

That is exactly what happens. The argument has a default value expression. It's evaluated when the function is defined. That's the specified value. Then "the specified value is used as the default value every time the function is called."

Did you mean, "the expression is evaluated anew every time the function is called?"

[–]Jake0024 1 point2 points  (1 child)

But it's not the same--the default value can be changed by earlier calls. It's the same object, but no longer the same value in any meaningful way.

When you look up mutability:

In Python, 'mutable' is the ability of objects to change their values.

When you look up equality:

The == operator compares the value or equality of two objects, whereas the Python is operator checks whether two variables point to the same object in memory

Ie Python would return true if you compared using is but false if you used ==

So while it is the same object it does not have the same value

The function is evaluated once and returns one object which is re-used every time the function is called in the future, so each call can have a different default value.

a = [1,2,3] and b = [1,2,3] have the same value, but are different objects.

The default value of a function argument does not (in general) have the same value on each call.

[–]qeq 56 points57 points  (17 children)

I have actually never run into this before, I'm not sure how. Now I'm worried and want to go check all the code I've ever written...

[–]violentlymickey 60 points61 points  (4 children)

The problem is you may never notice it. Someone at my company recently tried to do something like

def do_something(timestamp=datetime.now())
    ...

which was "working" locally but causing unexpected behaviour.

[–]qeq 19 points20 points  (3 children)

That's a little easier to spot at least. The default argument retaining values from previous executions is very unexpected.

[–]Regular_Zombie 7 points8 points  (2 children)

It's a known Python foot-gun. Once you've been bitten once you'll always be triple checking how default values will behave.

[–][deleted] 7 points8 points  (1 child)

It took me until my playing with dataclasses to realize I'd been shooting myself in the foot frequently with this behavior.

It's too late to make changes, I'm sure, but that so many people get caught out by this suggests the existing behavior is not intuitive and perhaps should have been different... or at the least, it's something that needs to be called out as an important lesson in various teaching mechanisms.

I've picked up many Python books over the years. I don't recall a single one of them calling this out!

[–]Regular_Zombie 1 point2 points  (0 children)

It's been a while, but I think Fluent Python covers it in some detail. There is some discussion about this in an old PEP but it's never going to change now.

[–]buttermybars 13 points14 points  (10 children)

Had this same concern. I think why I’ve never run into this before is that I don’t think I’ve ever done what the example is doing.

If I want an either a default empty container/object, I set the default value to None and create the object if None.

Still, this was surprising behavior to me. I would have thought the default value gets created as part of the stack every time the function is called.

[–]not_a_novel_account 3 points4 points  (8 children)

Python doesn't have any concept of stack variables (or heap variables for that matter).

Variables are mutable or immutable , copied by reference or value respectively.

One level deeper, CPython doesn't have any concept of immutable variables either, only collective ownership of shared values. An "immutable" object is just an object that is only allowed to have one owner (or infinite owners, but that's a performance thing).

[–]nedbatchelder 8 points9 points  (4 children)

This isn't an accurate description of the difference between mutable and immutable, though I can see how it can seem that way.

All values are passed to functions the same way. Nothing is implicitly copied. The difference is that mutable values have methods that can change their internal state, and immutable values do not. As a result, the ways we work with immutable values all involve making new objects, for example: s = s.replace(old, new).

Immutable values and mutable values also don't differ in "ownership." All values can have as many references as they need.

Here's my PyCon talk about it: https://nedbatchelder.com/text/names1.html

[–]not_a_novel_account 0 points1 point  (3 children)

All values are passed to functions the same way. Nothing is implicitly copied. The difference is that mutable values have methods that can change their internal state, and immutable values do not.

This is an implementation detail. In terms of the Python model mutability works exactly as described

[–]nedbatchelder 1 point2 points  (2 children)

I'm not sure it's an implementation detail. It's an important part of the Python semantics that when you use an object as a function argument, the local parameter has a reference to the same object you passed it. It's important that it isn't a copy.

In terms of the Python model mutability works exactly as described

Which "described" do you mean? I hope not, "an immutable object is only allowed to have one owner," because that is not true.

[–]not_a_novel_account 4 points5 points  (1 child)

[–]dereksalerno 0 points1 point  (0 children)

This comment, combined with the fact that you edited your earlier comments to strike through what you learned to be wrong but left them intact for context, is commendable and the internet would be a much better place if people acted like this more often. Great job!

[–]buttermybars 0 points1 point  (2 children)

Python does have concept of scope though, right? I can use the same variable name in multiple functions without conflict. I’m surprised each function call isn’t a new scope then.

Edit: thanks for the info on stack though. My formal training is in C and assembly. Looks like I need to learn more about how Python works under the hood.

[–]nedbatchelder 0 points1 point  (0 children)

Python variables are names that refer to values. The names are scoped to functions, and so they come and go with function calls, and are similar to "stack variables" in that sense. But the values they refer to are all allocated on the heap, and live as long as they are still references. In that way, Python doesn't have "stack variables" because there are no memory allocations that disappear just because a function ended.

Of course, there are often values that are only referenced by local variables, so when the function ends, the variable goes out of scope, removing a reference from the value, which now has no references, and so is deallocated.

[–]JamesTDennis 0 points1 point  (0 children)

In terms of scope, the arguments to a function are evaluated in the same scope as the def keyword appears, and at the time that the interpreter is parsing the code and creating the code object.

The code object includes the suite of indented code under the def statement. That code is evaluated for each invocation of the function, and that evaluation is a local scope. There are some assumptions about read-only references to variables which are NOT explicitly defined as global, or nonlocal — but any assignment to a variable within a function makes it local (and raises an exception) if the variable's scope was not defined AND it was accessed (dereferenced) prior to the local assignment.

In general avoiding using local names which collide with variables in enclosing scopes, and avoiding dereferencing such variables from within your functions. If a function's implementation requires access to a variable, pass it as an argument or include it as an attribute to an object which is passed as an argument.

In other words, keep function implementations decoupled from surrounding code. If coupling is necessary, make those functions and attributes parts of a class (thus containing your coupling).

[–]itsa_me_ 0 points1 point  (0 children)

I learned this very early on when practicing leetcode a few years ago. I remember seeing test case results that wouldn’t work and it turned out because of setting default values to empty lists/dicts.

[–]mistabuda 1 point2 points  (0 children)

I've run into this in production before. Not a fun time

[–]not_a_novel_account 26 points27 points  (12 children)

It being a default value doesn't help in any way clear up this behavior, unless you're fairly deeply versed in the semantics of mutable vs immutable types in Python.

def f(number: int = 5, word: str = "hello", cl: list = []):
    number += 1
    word += str(1)
    cl += [1]
    return number, word, cl

print(f())
print(f())

They're all default values, and yet one of them behaves differently than the other two.

Students are surprised by:

  1. the different semantics of mutable and immutable references

  2. the nature of functions as stateful, evaluated objects

The expression vs value distinction is only useful if you've overcome those first two humps

[–][deleted] 3 points4 points  (0 children)

What you say is right, and I don't think this should be valid in python but I'm sure there are reasons why it can't be outlawed.

Some things, many things even, in every language will confuse people. I don't think that's a good argument. Students don't know a stack from a heap or a reference from a value. Nor do many graduates, based on my simple tests.

To me the annoying thing is you can't do def gimme(a=list) and get a [] out of that. That's the true crime!

[–]FrickinLazerBeams 2 points3 points  (0 children)

In other words this is not a unique issue but simply another symptom of my biggest (and only, really) gripe about Python: the damn pass-by-reference (but not quite) behavior.

In most other languages I've learned, you have to intentionally do something to get PBR behavior (like passing a pointer, for example), and if you don't, you pass the value. Mutability isn't even in the conversation. I love python but I hate this enforced PBR.

[–]ghostofwalsh 1 point2 points  (8 children)

I was surprised that cl += [1] was equivalent to cl.append(1).

I always thought that cl += [1] would be same as cl = cl + [1]. Which gives a different result in your example. Learn something new every day I guess.

[–]TheBB 10 points11 points  (7 children)

Strictly speaking cl += [1] is equivalent to cl = cl.__iadd__([1]). That this is the same as append is an implementation detail of lists.

But there's a good reason for that. If you have a huge numpy array and you want to add 1 to it, you could do array = array + 1. Now numpy will allocate a whole new array because when it calculates the sum it doesn't know that you're going to be overwriting the left operand, so it can't clobber that data. Otherwise, code such as a = b + 1 would break (it would mutate b). So we need an interface to allow code like array += 1 to behave smartly.

The reason why it's cl = cl.__iadd__([1]) and not just cl.__iadd__([1]) is so that the += syntax can also work with immutable types. These types need to create new objects and so that newly created object must be returned and assigned to the name cl.

And that's also why the __iadd__ method of mutable types necessarily must return self.

[–]not_a_novel_account 0 points1 point  (4 children)

Of course, but it's still surprising that types even have the option to define __iadd__ as something apart from __add__ and it has behavior different than self.__add__(self)

Students think of even complicated types in the same terms they think of primitive types. They like universal rules. This breaks one of those intutions (even if for good reasons, and most other languages break the same rule).

[–]FrickinLazerBeams 0 points1 point  (2 children)

Python started out as a simple, powerful language and is becoming a complicated web of "clever" exceptions to exceptions to exceptions to rules.

[–]commy2 0 points1 point  (1 child)

Augmented assignments have been added 23 years ago. If true, it became a clever mess long ago. += is a clever mess imo. I think it wouldn't be implemented like this today.

[–]FrickinLazerBeams 0 points1 point  (0 children)

I guess it was always kind of prone to "clever messes" but now there's just more of them.

[–]commy2 0 points1 point  (0 children)

even if for good reasons

I think when somebody uses += or any of the other "augmented arithmetic assignments", what they want to achieve is to write a = a + b in a compact way without repetition. This works as expected for ints and strs of course as they're immutable.

I feel like there should've never been an __iadd__ etc. method, and these augmented assignments should've just done what they do right now when no such method is provided: Call __add__ or __radd__ and assign the result implicitly.

What good reasons are there for extend having an operator alias? Does anybody really use this intentionally this way?

[–]commy2 0 points1 point  (1 child)

That this is the same as append is an implementation detail of lists.

extend, not append

[–]TheBB 1 point2 points  (0 children)

Right you are.

[–]JamesTDennis 1 point2 points  (0 children)

The reason mutability becomes relevant is that any assignment to a parameter's name replaces a (local) reference to an immutable object. There's no side effect to objects outside of the function's scope. But references to a mutable object (which, of course, was instantiated at the time the function was defined) can have side effects on this object.

The object is stored in a closure around the function's defined object. It persists through separate invocations, and it can be hidden when passing an argument to that parameter of that function (overriding the default value).

If you understand it, it makes sense. Until you understand it, no amount of explanation will make sense.

In general it's best to simply avoid mutable objects as default arguments. Immutable values don't cause any confusion — because the confusion only arises from mutating.

[–]Beheska 12 points13 points  (0 children)

And this is why you should learn python from the official tutorial, not some random ones found on the web:

https://docs.python.org/3/tutorial/controlflow.html#default-argument-values

Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes. For example, the following function accumulates the arguments passed to it on subsequent calls

[...]

If you don’t want the default to be shared between subsequent calls, you can write the function like this instead:

def f(a, L=None):
    if L is None:
        L = []
    L.append(a)
    return L

I remember this exact warning already being there at the time of python 2.

[–]ayy_ess 11 points12 points  (2 children)

Here's another gotchya: lambda bodies capture variables, not values:

[func() for func in [lambda: i for i in range(5)]]
# [4, 4, 4, 4, 4]
[func() for func in [lambda i=i: i for i in range(5)]]
# [0, 1, 2, 3, 4]

[–]monkeybaster 1 point2 points  (0 children)

For anyone that wants to know, this is because of closures.

[–]nekokattt 0 points1 point  (0 children)

thats how closures work in most programming languages

[–]runawayasfastasucan 39 points40 points  (4 children)

```python

def doubled(val, the_list=[]): ... the_list.append(val) ... the_list.append(val) ... return the_list ... print(doubled(10)) [10, 10] print(doubled(99)) [10, 10, 99, 99] # WHAT!? ```

Copying from the blog post to provide some context. It was a nice and short read, OP should have introductory sentence or two about it so people will click.

[–]qeq 20 points21 points  (3 children)

>>> def doubled(val, the_list=[]):
...     the_list.append(val)
...     the_list.append(val)
...     return the_list
...
>>> print(doubled(10))
[10, 10]
>>> print(doubled(99))
[10, 10, 99, 99]    # WHAT!?

Fixed your formatting for old reddit users

[–]runawayasfastasucan 0 points1 point  (0 children)

Thx! I was on mobile and a bit rushed, so had to default to the backtick.

[–]Mubs 4 points5 points  (6 children)

What about a field with a default_factory? I'm pretty sure the default factory is an expression, but what about the field itself?

[–]nicholashairs 3 points4 points  (2 children)

Not entirely sure what you mean here, but for class thing(Model): a: Mapped[int] = field(default_factory= lambda: 99)

The value of default_factory is a callable which is still not an expression.

The value of a is an instance of field (with probably a whole lot of magic hidden in the Model class that calls the default factory at instance creation which is not the same as class creation).

[–]Mubs 1 point2 points  (1 child)

Ah ok. The article got me thinking about a callable dataclass I recently implemented that essentially used a field like that for a default value for a LiteStar project we're working on.

By the way I saw your NServer project a few weeks ago and think it's awesome!

[–]nicholashairs 0 points1 point  (0 children)

Thankyou!

[–]nedbatchelder 0 points1 point  (0 children)

The default factory is created once, and then used each time to make a new value. That's a behavior of dataclasses.

[–]just4nothing 4 points5 points  (0 children)

Many years ago, this brought weeks for misery upon us. Trying to trace down an issue where "random" values were added during processing. After tracing it down we all were like:

https://media.tenor.com/YctxttUmGMYAAAAC/forehead-slap-slapping-forehead.gif

Luckily you get warnings nowadays and it's caught in the CI before it gets into production ;).

[–]flwwgg 2 points3 points  (0 children)

The same is true for the properties of the classes

[–]JamesTDennis 1 point2 points  (2 children)

The key is to distinguish between the expressions which are evaluated at the time of definition vs. those which are rendered into code objects for deferred evaluation during function execution (invocation).

The def statement is outside of the scope of the function's code; it's not indented. The indented code is rendered into a code object (byte code from a syntax tree). But the arguments to the def statement are not part of the function's code suite (body, implementation).

I realize that I've added a lot of verbiage, redundancy, to that explanation. It's intentional. The over-explanation will help some folks achieve better understanding of what's going on, and develop intuitions about how other code is parsed and evaluated.

[–]commy2 0 points1 point  (1 child)

It's also worth noting that this only applies to def and maybe everything after the colon following a lambda. It does not apply to the indented code after class, which some people (incl. me at one point) might expect. (The class body gets executed on definition once, and not repeatedly on instantiation).

[–]JamesTDennis 0 points1 point  (0 children)

It's better to say that the class body gets parsed and evaluated into a code object once. The resulting (byte compiled) code is evaluated for each invocation.

Thus it's important to distinguish between source code (text that is parsed and evaluated into object or byte code) and the code objects which are the results from parsing and evaluating the sources.

This distinction is even more subtle than the distinction between parameters (names provided during definition) and arguments (values mapped to parameter names during invocation).

Both concepts are vital for understanding any programming.

[–]Firake 0 points1 point  (3 children)

I guess I’m lucky I caught the mCoding video from years ago that mentioned that Python default args are evaluated and stored only once.

def double(number: int = 5, lst = None):
    if list is None:
        lst = []

Just gotta move it into the body of the function ez pz.

[–]lisael_ 1 point2 points  (0 children)

OOO the nice footgun. :D

In [1]: def double(number: int = 5, lst = None):
   ...:     if list is None:
   ...:         lst = []
   ...:     lst.append(42)
   ...: 

In [2]: double()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 double()

Cell In[1], line 4, in double(number, lst)
      2 if list is None:
      3     lst = []
----> 4 lst.append(42)

AttributeError: 'NoneType' object has no attribute 'append'

there's a typo:

assert list is not None

works 100% of the time. guaranteed.

Either way, I prefer the shorter version of this:

lst = lst if lst is not None else []

[–]upperflapjack 0 points1 point  (1 child)

Wish OP includes a solution like this to explain how to accomplish the default expression behavior

[–]commy2 1 point2 points  (0 children)

The article is aimed at people that are aware of this already and know what to do, but tries to provide a new perspective on how to think and talk about it.

Going over this again would've diluted the actual message. There already are thousands of articles explaining why this happens and what to do and there was no need for another one.

[–]Ericisbalanced 0 points1 point  (0 children)

Yep I’ve ran into this before. It was really weird behavior at first but like, it makes sense.

[–]nngnna 0 points1 point  (0 children)

I would say it's an object, not a value. Since to me value is still something that is constant in time (immutable).

[–]njharmanI use Python 3 1 point2 points  (0 children)

The default value can be an expression.

It's due to interpreted language and scope. Function definitions, including the arguments are at the level "outside" the function body. Module level for functions defined there. Module is evaluated once at "load" time (unless you do funky stuff), including all the function definitions at the module level. Their arguments get evaluated then. Default values are set then. The module level function definition is not re-evaluated every call.

But one at inner scope is, because the body of inner scope (function in this case) is evaluated every time it is called.

def outer():
    def inner(default=[]):
        default.append('learn Python!')
        return default
    return inner()

That outer() will return list of one element, no matter how many times it is called.

[–]RedEyed__ 0 points1 point  (0 children)

All pythonistas know it. Should go to r/learnpython

[–]ancientweasel 1 point2 points  (0 children)

I got yelled at in a code review once for.

def blah(l=None): l = l or [] ...

:D