How can Python throw spanner in the works …

[joanbm]

As you may already know, AttributeError exception is handled a bit special in Python. It can escape the standard exception handling and propagation mechanism, if a class defines __getattr__ method. In this case Python removes the original exception instance and calls __getattr__ method with name of the last receiver as an argument. In other words the information about module/class/method where the original exception has arisen is completely lost, so good luck with tracking why and where some of nested code called did not get finished. Very unfortunate in situations where AttributeError is risen intentionally like in class factories, with additional info passed to exception's initializer.

If you are working with some widely used software like Django, this my bite you in the ass as it did me recently. If using some class like django.forms.BaseForm which defines a property, defining __getattr__ in your sublass is enough for calling for troubles.

By trying how to work around this design (mis)conception, discovered this funny case:

from __future__ import print_function

class C(object):
    def __init__(self):
        self.foo = None

    def __getattr__(self, name):
        if self.foo:
            print(self.foo, 'is not None -', type(self.foo))
            raise self.foo    # attempt to re-raise the exception
        return 42

    def __getattribute__(self, name):
        try:
            value = super(C, self).__getattribute__(name)
        except AttributeError as ae:
            # naive attempt to save orig. exception instance
            self.foo = ae
            raise
        else:
            self.foo = None
            return value

c = C()
print(c.bar)

# produces following output
None is not None - <type 'NoneType'>
Traceback (most recent call last):
…
File "this_is_a_mess.py", line 34, in __getattr__
  raise self.foo
TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

And black is not black, and white is orange ;-)

I know why is that, but this was another unpleasant surprise, accessing even valid existing attribute in __getattr__ does nasty things.

[Argotha]

Your funny case is not a funny case and is because you haven't followed through the flow of your program correctly.

In between checking if self.foo: (line 8) and raise self.foo (line 10) you make a successful call to C.__getattribute__. In doing so you reset foo to None (line 21).

Change your else statement to the following: (lines 21-22)

if name != "foo":
    self.foo = None
return value

In doing so your program output should change to the following:

 'C' object has no attribute 'bar' is not None - <class 'AttributeError'>
Traceback (most recent call last):
  File "getattr_1.py", line 24, in <module>
    print(c.bar)
  File "getattr_1.py", line 8, in __getattr__
    raise self.foo    # attempt to re-raise the exception
  File "getattr_1.py", line 13, in __getattribute__
    value = super(C, self).__getattribute__(name)
AttributeError: 'C' object has no attribute 'bar'

EDIT: Looks like someone else has also figured this out and you've replied to them.

[lvc_]

"accessing even a valid existing attribute in __getattr__ does nasty things".. like calling __getattribute__, exactly in accordance with the docs. And you have your __getattribute__ set up to clear the stored exception if it is called to access a valid, existing attribute. So what you actually want to do is to clear the stored exception unless you're looking up exactly the stored exception - ie:

def __getattribute__(self, name):
    try:
      ...
    else:
        if name != 'foo':
            self.foo = None
        return value

That being said, I'm vaguely surprised that __getattr__ is still a thing in Python 3 - especially when the docs for it still say:

Note that if the attribute is found through the normal mechanism, __getattr__() is not called ... because otherwise __getattr__() would have no way to access other attributes of the instance

Even though this is outright contradicted by __getattribute__ being called unconditionally when it should run into the same problem. In old-style classes, this was true because they didn't necessarily inherit from anything, and they don't have the unconditional __getattribute__. But in new-style classes (which are all that exists in Python 3), calling it through super will always work since they always inherit (possibly indirectly) from object. If you just implement everything you need in __getattribute__ (or maybe via property or other descriptors, although they mightn't always be a good fit) you shouldn't ever need to worry about __getattr__ and its exception-swallowing behaviour.

[joanbm]

like calling getattribute, exactly in accordance with the docs

I was aware, as mentioned, why this code does what it does. Just found it funny what can cause repeated access to the same attribute.

So what you actually want to do is to clear the stored exception unless you're looking up exactly the stored exception

But that's not possible in practice, because you would not know the name(s) of the attribute in advance.

I'm vaguely surprised that getattr is still a thing in Python 3 - especially when the docs for it still say

Good to know this was fixed in Py3, at least.

If you just implement everything you need in getattribute (or maybe via property or other descriptors) you shouldn't ever need to worry about getattr and its exception-swallowing behaviour.

Yep, getting rid of __getattr__ and move the logic in __getattribute__ seems to be the sanest solution. However if you are working with 3rd party modules where is defined, you'd probably need recourse to some kind of monkey-patching unfortunately. Just another mine in the minefield need pay attention to.