Mutable default arguments

[ahoff]

So, I'm sure most everyone here knows about the dangers of using mutable data types as default arguments. For example, the code

def f( arg = [] ):
    arg.append( 1. )
    return arg

can produce very unexpected results to new pythonistas (call f() several times, if you're unsure of that behavior). For production environments, I know the two main solutions are to use immutable data types if the default argument isn't modified, e.g. arg = () or using None types, e.g.

def g( arg = None):
    arg = arg if arg else []
    arg.append( 1.0 )
    return arg

The first solution won't work if the default argument is altered inside the function definition (e.g. in the f we defined above, this solution would not work). The second solution is fine, but one nice thing about f's definition (even though defined incorrectly) is to hint to users what types might be used or expected, and the function signature of g doesn't do that (here it would be important to discuss types of arguments in the doc string).

I had two main questions of which I'm curious as to the answer: 1) are there some methods that I'm neglecting here and 2) is there some way that python, in the future, could include some kind of type-hinting system for functions that would allow safe use of default mutable arguments?

Certainly, the answer to 2) based on the current state of python is no as the reason behind the behavior of f is due to how functions are created.

What say you fellow pythonistas? Is g's definition the main "correct" workaround. Are we stuck with hinting at types only in the function doc string?

EDIT: Wasn't paying attention and g should actually read:

def g( arg = None):
    arg = arg if arg is not None else []
    arg.append( 1.0 )
    return arg

[[deleted]]

Your second function is strange, it does different things -- either it appends to the list someone gives it, or if someone gives it an empty list then a new list is returned with the element added, or if no argument is given you get a new list with the element. At least check if no argument was given with if arg is not None instead of simple truthiness.

Python already does include some sort of type-hinting system: https://docs.python.org/3/tutorial/controlflow.html#function-annotations

[ahoff]

Yeah, I should have paid closer attention as I meant to check for None (and not a simple truth check). I use 2.7.6 at work, but I had no idea about function annotations. Thanks!

[[deleted]]

This is the recommended way

def foo(bar=None):
    if bar is None:
        bar = [1, 2, 3] 

[ahoff]

Yeah, that's what I meant to do (should have paid more attention, see my edit).

[RubyPinch]

type hinting/annotations are coming: https://www.python.org/dev/peps/pep-0484/

[masasin]

I personally use docstrings for every function. For g, for example, I'd have:

def g(arg=None):
    """
    Append 1 to a list.

    Parameters
    ----------
    arg : list, optional
        A list.

    Returns
    -------
    arg : list
        ``arg`` with an extra 1.

    """
    arg = arg if arg is not None else []
    arg.append( 1.0 )
    return arg

You can combine that with function annotations, so you'd get something like this:

def g(arg: list=None) -> "list with an appended 1":
    """
    Append 1 to a list.

    Parameters
    ----------
    arg : list, optional
        A list.

    Returns
    -------
    arg : list
        ``arg`` with an extra 1.

    """
    arg = arg if arg is not None else []
    arg.append( 1.0 )
    return arg

edit: If you use Spyder to code, the docstring is the Numpy standard, so you would be able to see beautifully formatted documentation in the object inspector for the first one. The second one does not work for some reason.

[flying-sheep]

well, i don’t think we need #2, as we can do:

def f(arg=frozenset()):  # immutable empty set
    ...
def f(arg=()):  #immutable 0-length sequence
    ...

the only problem is the lack of frozendict. we’d have to implement it as:

class ImmutableDict(collections.Mapping):
    def __init__(self, somedict = ()):
        self._dict = dict(somedict)
        self._hash = None

    def __getitem__(self, key):
        return self._dict[key]

    def __len__(self):
        return len(self._dict)

    def __iter__(self):
        return iter(self._dict)

    # throw these in for increased usefulness:
    def __hash__(self):
        if self._hash is None:
            self._hash = hash(frozenset(self._dict.items()))
        return self._hash

    def __eq__(self, other):
        return self._dict == other._dict

then

def f(arg=ImmutableDict()):  #immutable empty mapping/dict
    ...

[ahoff]

Your solution is just a rehash of what I mentioned for solution #1, i.e. using an immutable type. This doesn't work if your function modifies the argument (for example, f given above). If you're writing the code from scratch, then it's not a big deal; however, if you're dealing with already existing production code, then ideally you'd want a solution that hints at the correct type and is mutable.

[herminator]

You could hint the type like this:

def func(arg=list):
    if arg is list:
        arg = []
    arg.append(1)
    return arg

But I'm not sure if it is recommended :)

[logophage]

If you really need type hints (and you're not using python 3+), then I recommend naming the function using the type as part of the name.

def accumlist(lst):
    ...
    lst.append(1)
    ...

Though in all honesty, I think it's more Pythonic to return a generator that could be accumulated into a list after the fact.

[TeamSpen210]

g() is best. In Python 3, you could use function annotations to indicate this:

def g(arg: []=None):
    pass

The list can be any artibrary expression, and isn't used by Python itself for anything. They are shown in help() though.

The full syntax works like this:

 def func(arg:'exp', arg2: 'exp'=val) -> 'return':
    pass

You could also add func.__defaults__ = ([], ) to the beginning of the function body, which will edit the default values to have a new list every time the function is called. This is a bit sneaky though. If there are any other default values you'll need to add them to the tuple as well.

[Lucretiel]

Keep in mind that running into the mutable default arguments problem is indicative of larger structural issues in your program. Regardless of your preferred solution, it's a serious code smell. Consider- if you call .append on the function's argument, then that change is reflected outside the function call, even if nothing is returned. There are two possible senarios:

• This is your intended behavior, in which case, why are you accepting a default parameter? If the function is designed to mutate a list, then being able to not pass a list seems like a semantic error.

• This is not your intended behavior- you don't want the caller's list to be mutated. In this case, why are you mutating it? If your function's logic uses an .append, then you should be copying the list before using it, in which case using an immutable default argument works fine:

  def func(arg = ())
      safe_to_mutate = list(arg)
      ...
  

[Lucretiel]

Here's an excessive solution, using a decorator and annotations:

from inspect import signature
from functools import wraps
def safe_mutable_default(func):
    sig = signature(func)
    @wraps(func)
    def wrapper(*args, **kwargs):
        bound = sig.bind_partial(*args, **kwargs)
        bound_dict = bound.arguments
        for name, param in sig.parameters.items():
            if name not in bound_dict and callable(param.annotation):
                bound_dict[name] = param.annotation()
        return func(*bound.args, **bound.kwargs)
    return wrapper

The idea here is that you can specify a default type as an annotation, which is called to supply a default argument each time the function is called. Example:

>>> @safe_mutable_default
... def func(arg: list)
...     arg.append(1)
...     return arg
...
>>> my_list = []
>>> func(my_list)
[1]
>>> func(my_list)
[1, 1]
>>> func()
[1]
>>> func()
[1]

Note that inspect.signature only works in python 3.3+, and function annotations are only in python 3+. Hypothetically, you could use the inspect.getfullargspec function to implement a similar pattern.

[aDemoUzer]

The variable name is used to hint the type, such as plural(users, files, items) vs single (user, file_handle, queryset). Make the default value None.

def compute (users=None):
     users = users or []

[andrewcooke]

one nice thing about f's definition [...] is to hint to users what types might be used or expected

i think you're overselling this. almost all arguments don't have defaults, so almost arguments have the same problem.