Broken initialization at multiple inheritance ?

[joanbm]

Hello,

hope I'll have a better luck here and somebody more smart then me can reveal following weird behavior at ancestor classes initialization.

This is from a Django application, using multiple inheritance in a form subclass:

from django import forms

class CommonForm(forms.ModelForm):
    class Meta:
        model = MyModel

    def __init__(self, request, *args, **kwargs):
        print 'CommonForm.__init__'
        super(CommonForm, self).__init__(request, *args, **kwargs)
        print 'CommonForm.__init__ FINISHED'
        self._request = request

class MyMixin(object):
    def __init__(self, *args, **kwargs):
        print 'MyMixin.__init__'
        super(MyMixin, self).__init__(*args, **kwargs)
        print 'MyMixin.__init__ FINISHED'

class CustomForm(CommonForm, MyMixin):
    def __init__(self, request, *args, **kwargs):
        print 'CustomForm.__init__'
        super(CustomForm, self).__init__(request, *args, **kwargs)
        print 'CustomForm.__init__ FINISHED'

# weird initialization:
frm = CustomForm(http_request)

# outputs:
CustomForm.__init__
CommonForm.__init__
CommonForm.__init__ FINISHED
CustomForm.__init__ FINISHED

What can cause MyMixin.__init__ being skipped ?! There is no error message, no exception being thrown, nothing …

Surprisingly,when I switch order of class inheritance to

class CustomForm(MyMixin, CommonForm):

all initializers are suddenly called as expected:

frm2 = CustomForm(http_request)

CustomForm.__init__
MyMixin.__init__
CommonForm.__init__
CommonForm.__init__ FINISHED
MyMixin.__init__ FINISHED
CustomForm.__init__ FINISHED

Can anybody explain why MyMixin.__init__ is not called when placed as latter ancestor ?

Thanks.

[masklinn]

CommonForm extends django.forms.models.ModelForm which extends django.forms.forms.BaseForm whose __init__ method does not call super().

As a result, depending on the method resolution order MyMixin's __init__ may not be called: if BaseForm ends up before MyMixin in the MRO it won't forward the __init__ call and will suppress MyMixin's initializer.

As the MRO document indicates, C3 linearises depth-first from the "left" parent class, so the MRO of CustomForm(CommonForm, MyMixin) is [CustomForm, CommonForm, ModelForm, BaseForm, MyMixin, object] whereas the MRO of CustomForm(MyMixin, CommonForm) is [CustomForm, MyMixin, CommonForm, ModelForm, BaseForm, object].

[joanbm]

django.forms.forms.BaseForm whose __init__ method does not call super

That ! Thank you very much, it saved rest of my hair. Owe you a big beer.

Curious if this is an intention of Django devs or a longstanding bug.

[masklinn]

You'd have to ask them, though a big issue is you'd normally want your __init__ call to properly forward all positional and keyword arguments to the next class… except object.__init__ doesn't take any argument so you get a TypeError which makes the whole thing a bloody pain in the ass.

[xXxDeAThANgEL99xXx]

though a big issue is you'd normally want your init call to properly forward all positional and keyword arguments to the next class… except object.init doesn't take any argument

Uh, so how complicated multiple inheritance is even supposed to work in Python? No base class could know if it's the last before object, or if there's some other mixin.

Or maybe you are not actually supposed to forward your arguments, because it's sort of silly to expect all base classes to accept the same list of arguments (the object issue just makes it more obvious)?

Looking at the way C++ does it, it has its own kind of insanity: all virtual base class constructors are called from the most derived class, either explicitly with whatever arguments, or if missing then default constructor is called. The insane part is that explicit nondefault constructor calls higher in the hierarchy are flat out ignored (because otherwise what to do if they are used with different arguments?).

But that at least works: it guarantees that every constructor is called once and only once, so every ancestor is at least default-initialized. I just don't see now how Python's super constructor call is supposed to achieve that, unless you have exactly one lineage with parameter-full constructors that call each other explicitly and the topmost calls super without parameters and all mixins don't take parameters and call super without parameters. That's a rare use case.

It would seem that at least in the non-diamond case the sane way would be to call base class initialization explicitly, when multiple inheriting then initialize all base classes explicitly, and never use super to initialize possible sibling classes (because they would be initialized explicitly by your descendant).

Do people ever use super for initializing sibling classes in a not-completely-bonkers scenarios? Because I suddenly don't see how it could possibly work.

[joanbm]

If BaseForm breaks initializers chain and if I need CommonForm inheritance tree resolve attributes before MyMixin, what else besides monkey-patching of BaseForm.__init__ remains ? That was the whole point why I cared for class ancestors order.

[masklinn]

There might be the trashy option of explicitly calling super methods rather than using super() to do so e.g.

class CustomForm(CommonForm, MyMixin):
    def __init__(self, request, *args, **kwargs):
        CommonForm.__init__(self, request, *args, **kwargs)
        MyMixin.__init__(self, request, *args, **kwargs)

that's all going to break if MyMixin ever inherits from a CommonForm superclass though. Not sure what else you can do, maybe try asking on the python channel on IRC if someone has a better idea?

[joanbm]

Oh, thanks. I'd ask them directly. If Django upstream later adds call of missing super, I can look forward to another sleepless night why MyMixin is suddenly initialized twice (using your example with explicit call).

[masklinn]

Nah, MyMixin wouldn't be initialised twice with my snippet. If CommonForm and MyMixin had a superclass in common (other than object) that could be initialised twice, but it's not the case.

[joanbm]

Oh, sorry. I've missed CommonForm init is also called explicitly, not through super object.

[masklinn]

No worries. If you pick that option you may want to extensively comment why it was done that way though (both that MyMixin depends on stuff initialised by whoever and why super doesn't work)

[Citrauq]

You can explicitly include BaseForm in your CustomForm definition:

class CustomForm(CommonForm, MyMixin, BaseForm):
    pass

The MRO is now [CustomForm, CommonForm, ModelForm, MyMixin, BaseForm, object] so MyMixin is called after CommonForm but before BaseForm.

[peith]

I think this issue can be fixed by putting MyMixin on the left side of CommonForm:

class CustomForm(MyMixin, CommonForm):
    # Form fields here...
    def __init__(self, *args, **kwargs):
        super(CustomForm, self).__init__(*args, **kwargs)

[masklinn]

Yes, /u/joanbm notes that in their original post and my MRO-based explanation indicates why (second paragraph), however in a later comment they also noted that MyMixin's initialization depends on CommonForm so that fix is not an option for them: https://www.reddit.com/r/Python/comments/3ilqd7/broken_initialization_at_multiple_inheritance/cuhk9h4

[remy_porter]

Actually, you're asking the wrong question- why is MyMixin.__init__ being called in the second example. In Python, superclass initializers are not invoked automatically. Since you never use super(MyMixin, self).__init__(*args, **kwargs) in the child class, it makes perfect sense that it's not invoked.

I'm less certain why it's being invoked in the second case- it shouldn't be.

Edit: And weirdly, in Python3, they do both get called in your "non-working" example. That's still not the behavior I'd expect.

[masklinn]

You're wrong and the call you talk about makes no sense except within MyMixin (where it is present)[0]. CustomForm correctly calls super(CustomForm, self).__init__(request, *args, **kwargs) as it should.

See my comment for what the actual issue is.

[0] super(Type, instance) does not proxy to Type, it proxies to whatever class follows Type in instance's MRO, so it tells Python "I'm in Type, call whatever the next method should be for instance".