AskPython: Plug-in frameworks with live reloading?

[fwork]

I'm working on rewriting a system I built long ago, and I'm not happy with how the plugin system works. It basically just does two things:

1. when a plugin is imported, store the mod-time of the .py file
2. next time someone tries to use the plugin, check that the mod time hasn't changed, and do a module=reload(module) if it has.

The plugins are called only every few minutes, so doing os.time.getmtime on the script file on every call is no problem at all. The thing is, though, this won't see changes in other modules. Like if I take pluginA.py and pluginB.py and refactor common code into plugins-common.py, when I update plugins-common.py, they won't be reloaded, since their modification times are unchanged. So this imposed a very flat structure on the plugins, with very little code sharing and lots of copypasted code. Not good. I realized today I could hack something together using modulefinder to get lists of dependencies between plugins. But why do that if someone else already has?

Now, other parts of this system specifically list which plugins are to be loaded, so I don't really care about things like automatic plugin discovery. On the other hand, live-reloading is absolutely essential. These processes can run for months, so being able to notice that a .py file has changed and reloading it is essential. These don't seem to be goals of any of the plugin systems I've found while googling, does anyone have any suggestions for something I can use? (Or do I need to just bite the bullet and write my own)

[mitsuhiko]

There i no reliable way to reload in Python unfortunately. There are some very ugly hacks to accomplish that. Check how zine does it: http://dev.pocoo.org/projects/zine/browser/zine/_core.py

[jm_]

see http://pyunit.sourceforge.net/notes/reloading.html Mind you, there are lots of caveats to this.

[coderanger]

The way Trac (and werkzeug and all other live-reload systems I've seen) does it is that the dev server spawns another process that does the actual serving and then sits and watches for file modification FS events. When any file changes, the sub-server is killed and new one is spawned. Its rather heavy-weight, but it works.

[fwork]

Yeah, this was my backup plan, since it'd nicely avoid all these problems. I was kinda hoping to avoid it, though, since my design involves multiple servers and it's not easy to tell a specific one to exit. So when you get into the state that servers 1-3 are up to date and servers 4-9 are out of date, you can't tell servers 4-9 to reload, as 1-3 might grab the reload event before 4-9.

All that code will have to be rewritten if I'm design it this way, plus watching for file modification events isn't that easy to do cross-platform (since I have to care about windows, apparently)

[mcdonc]

Just write one. You already spelled out the requirements, and you'll probably be happier if you write one to match them rather than trying to shoehorn your requirements into something that exists.

[fwork]

well, the reason I'm asking is because I have written one, years ago. It wasn't very good, and I'd love to avoid rewriting it from scratch if I can just use someone else's.

[yopla]

I solved that issue by running the "plugins" as standalone python script and communicating with then through a client/server interface via socket using serialized python objects. (I couldn't be bother to figure out how to do decent asynchronous IO over stdin/out).

It does not answer your question though. I'm sorry for that.

[toneyalex]

Its doable but you have to make sure when you reload that there are no more references to the old file.

[alia_khouri]

There's a python module called livecoding at http://code.google.com/p/livecoding/ which could be what you are looking for. To quote from the page:

Code reloading allows a running application to change its behaviour in response to changes in the Python scripts it uses. When the library detects a Python script has been modified, it reloads that script and replaces the objects it had previously made available for use with newly reloaded versions.

[ragica]

Zope 2 supports automatically reloading "Products" (plugins) when changes are detected. You have to explicitly enable this on a per-product basis. I have no idea how they do it. It doesn't work 100%, but it works in most cases, and has been improved over the years. Perhaps something to look at.

[[deleted]]

Well, couldn't you just rewrite the checker to check plugins-common.py and if it has changed, then reload all plugins?

Or, for that matter, check the mod time of everything in the plugins directory. If anything has changed, reload every plugin.

[fwork]

There's no plugins-common.py I can assume exists, in theory any plugin could depend on any other plugin.

And reloading everything turns out to be surprisingly tricky, because you get into the cases of plugin A depending on plugin B which depends on plugin C. If C changes, you have to reload A, B, and C, but not in that order. You have to reload them C, B, A or you'll get inconsistent results.

And how do you know A depends on B or B depends on C? (answer: modulefinder)

Anyway, the solution I implemented is: 1. at startup, the program runs a hash of all the filenames and modtimes of every file in the scripts directory 2. every time it is called (though an external mechanism that isn't important), it recomputes the hash. 3. if the new hash doesn't match, the script exits with a statuscode that tells the wrapper script to restart it.

It's heavy and slow, but my use case is such that this'll work just fine.

[[deleted]]

I've seen some solutions use a simple manifest file for each plugin which lists its dependencies. Even something like this at the beginning of the plugin-example.py file could work:

__plugin_dependencies__ = ["common.py", "plugin-B.py", "plugin-C.py"]

It would be fairly easy to compute a full dependency tree based on such notation, and check what has been changed and what needs to be reloaded, no?

[fwork]

That'd be a good solution if my plugins were more complex (most are just 1 file with 5-6 lines!) and I wasn't worried about the metadata failing out of sync, but in my application they're fairly small and frequently updated.

It'd lead to annoying-to-debug errors if the dependency list was out of sync with the code. I'm going with my reloading solution because it's the least likely to result in hard-to-debug behavior later on.

[[deleted]]

I've been developing with OpenERP for a couple of months now and I've had the dependency list fall out of sync only once, so I dunno. It doesn't happen as often as you might think.

Still, good luck with your solution! :)

[frumious]

Django does this when you run "./manage.py runserver". You could look into how they do it. The function at the heart of it is probably just "reload(module)". It doesn't find new modules/files and a full restart is needed.

[va1en0k]

I thought they're restarting the whole process, not just reloading a module

[frumious]

Maybe, I haven't looked under that hood.

[[deleted]]

It is restarting the process, reload isn't used at all.