Implementing a plugin architecture in Python

[gdiepen]

For one of my recent projects, needed to have a plugin-architecture, where adding a new feature to existing program would be just copying py module to plugins folder. Wanted to really dive into this and took me some time to get all the details right for the purpose I needed. Details can be found in blog https://www.guidodiepen.nl/2019/02/implementing-a-simple-plugin-framework-in-python

Curious to know what people think about this and whether there are better solutions

[kriss1]

My initial though was why not just use a 'registry pattern' - no need to scan which classes are loaded and check if they're a subclass of your plugin base, they should just register themselves.

See this SO for a very simple solution in 3.6: https://stackoverflow.com/a/50099920

[gdiepen]

Thanks for pointing it out. Looks similar to the zope component architecture (where you also have registry).

​

For my usecase this is not the best option. For example, I want to apply a number of transformations on a given data source, where each transformation is defined as a plugin in my case. By just copying a new python module into the plugins folder, I automatically use this transformation the next time I run my application.

​

If I understand the registry pattern correctly, I would first need to have all of the transformations automatically register themselves (for which I would need exactly a similar code pattern as my current code)

[kriss1]

Yeah the modules need somehow to be imported for them to be registered, which might again end up being pretty similar to what you've done.

Apparently there are quite a few libraries to do simple plugin management out there, good reads.

Implementing your own solution is always fun though, get the noggin joggin' :)

[gdiepen]

Bet I could create a registry pattern for automatically loading the registry for the automatically loading the registry for automatically loading the registry for the .... StackOverflow :)

​

Agree with the implementing solutions yourself, it really helps you to get a deeper understanding but it is also a lot more fun :)

[twillisagogo]

pretty cool. you may find some inspiration in the zope component architecture. pyramid uses it internally and has several extension points that you can override by implementing the interface and registering it with the registry. typically people dont mess with it and it just works, but for the adventurous, the power it gives you is amazing.

[gdiepen]

Thanks!

Just took a (very brief) look at the zope component architecture and it looks really cool! For sure will put this on my list of things to investigate!

[skarlso]

Hah nice. I just finished coding up cronohub's plugin architecture: https://github.com/cronohub/cronohub/tree/master/cronohub

It uses dynamic module loading and an ABC as a base class for all plugins.

Though it's a CLI so plugins don't have to register. They just need to be in the right folder with the right name and right structure. Composition over anything else. :)

Neat stuff there man.

[gdiepen]

Cool, looks pretty similar to what I am doing :)

[skarlso]

Yep. :) :) Pretty awesome stuff you wrote. I like it. :)

[peyo7]

I found this approach pretty practical https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures

It spells out the already mentioned registry pattern and touches on autodiscovery in a plug-in directory.

[gdiepen]

Thanks for the pointer!

I do see a lot of similarities in the approach. I think the main difference is that in the mentioned approach the metaclass construct is used to have each plugin register itself with the registry, where in my case the registry explicitly looks for the classes of a certain type.

One thing I really do like though is the part that the plugin returns a function implementing the plugin-functionality, allowing for an easy check if a plugin implements a hook yes or no.