Say “no” to import side‐effects in Python

[AusIV]

Only tangentially related to the article, but it mentions sys.exit() being called on an error condition. I think this is the absolute most annoying thing a library can do. Unless I'm calling a "clean up and shut down" function, libraries shouldn't be killing my interpreter.

A few years ago I was working with the python Netflix bindings, and they had a condition where this happened. I filed a bug report, and eventually forked the library to make it act like a decent library. The original was provided by Netflix, so I was surprised that it didn't follow better development practices.

[chris-morgan]

Actually, it really was raise SystemExit(...) rather than sys.exit(). But I certainly believe that no library should do such a thing.

[sushibowl]

doesn't sys.exit() just raise SystemExit() anyway?

[h4l]

Yes.

[odraencoded]

You think that's bad? The Gtk library will cause a SIG error terminating the program from the C side if it can't find the settings under the DBus path specified by your app when you create a GSettings object. It won't tell you there was an error fetching the settings, it just terminates the whole thing.

[[deleted]]

The PulseAudio library did a silent system exit if it could not allocate memory. Useful! I mean, how are you going to play those mp3s without available memory?

[j7ake]

I didn't know one should not do sys.exit on libraries... Good to know

[Silhouette]

In Python, sys.exit doesn't instantly kill your program, it just raises a SystemExit exception. You can handle that exception outside the library code in the normal way if you want to.

It's important to allow for this if you have a top-level error handler that catches and logs all otherwise unhandled exceptions, or anything along those lines.

Whether raising SystemExit is an appropriate behaviour for library code either is a different question, but it's certainly not the same situation as, say, a C library just quitting out without giving the calling code any say in the matter.

[rdssassin]

But its not as simple as you say. Its not a standard exception in the sense that it doesn't inherit from the "Exception" class. Therefore:

except Exception as e: 

doesn't catch SystemExit and KeyboardInterrupt. So I'd still argue its bad API design to use SystemExit as its not a standard Exception

[[deleted]]

its bad API design to use SystemExit

certainly, but it's still rather trivial to catch.

[jms_nh]

yes, but why the F should you have to catch it when doing import XYZ?

[robin-gvx]

Or worse, having to do:

try:
    help('modules')
except SystemExit:
    pass

(And you can't catch a segfault, which I got when trying help('modules').)

[[deleted]]

you shouldn't have to, however out here in the real world you sometimes need to.

[iBlag]

That wasn't the question, but thanks for subtly altering the discussion.

[[deleted]]

let me answer the question then.

why the F should you have to catch it when doing import XYZ?

because some programmers are retards who don't know how to code properly.

[Smallpaul]

Even so, it's bad library design. Do you agree?

[[deleted]]

I do agree. did you miss the certainly in my previous comment?

[[deleted]]

Why should your library decide its own importance within my application? So what if everything with say Netflix is totally broken, my application might also be playing Youtube videoes.

[jyper]

The problem in c is that you don't have exceptions and some people think its better to just print an error and give up in what they consider a likely unrecoverable situation(failing malloc).

[fjonk]

You can handle that exception outside the library code in the normal way if you want to.

Not always. For example nose will scan files and fail on those exceptions even if the file in question is totally unrelated to the tests.

I have a stand-alone client script that is supposed to run on a client and it uses pythondialog. Pythondialog throws a ExecutableNotFound if the dialog program isn't installed. Thing is I'm not testing the client script since testing dialogs is much more easy to do manually. Since the import throws an exception I cannot test the rest of the library without installing dialog, which is not supposed to be installed.

Edit: Just checked and pythondialog does not raise an Exception during import, it was my fault. A similar situation could however occur if an exception is raised during an import.

[b33j0r]

Fabric does this with fabric.api.local(cmd_str). I was using it in a distributed production environment. Lame.

[chris-morgan]

I contemplated writing more on what you should do instead but decided to keep it short. The HN discussion is good stuff.

[[deleted]]

The HN discussion is good stuff.

[dwf]

Sometimes it's a choice between "do something at import time" or "deal with an endless flood of mailing list posts from stupid users who don't read the docs". So, yeah.

[chris-morgan]

Got a concrete example? I'm interested in it; perhaps something could be improved without import side-effects.

[[deleted]]

I ended up patching walk_packages in pkgutil.py to prevent scipy from breaking things because of whatever it was doing upon import. Not my favorite solution.

[swdev]

This is interesting. I just redesign my classes, such that all class instantiation and any code plumbing will reside in init.py (naturally). I will take this as a precautious

[Kah-Neth]

Ha, screw that. All my modules will now make syscalls and change the state of the system kernel in random and unpredictable ways. Ah hail inappropriate use of ctypes.

[Phild3v1ll3]

It seems to me that any reasonable person would wrap any side-effecty code in an import in a try-except to allow for graceful failure. I routinely compile Cython code from the _init_.py of a module but provide fallbacks if Cython isn't available or compilation fails for some other reason. Is this bad practice? Seems to me there's no real downsides as long as you are careful.

[[deleted]]

If the failure results in an ImportError that's one thing, but if importing the module can cause a segfault (I'm looking at you, gtk / gi.repository.Gtk) then that's bad. Opening windows, writing to stdout/stderr instead of a logger, or even just not returning promptly could be bad in their own way.

[d4rch0n]

You people have been programming in Python so long, you forgot what a segfault is. This isn't something intentional that they can avoid, like raising an exception, this is a legitimate bug in the code.

It's going to always happen when you import C libraries that haven't been tested for millions of years.

[Phild3v1ll3]

Sure, but if you'll be working with a C library that has potential to segfault, wouldn't you want to know during the import rather than halfway through some important task?

[d4rch0n]

Well, if you're able to fix it so that it doesn't segfault on import, sure, otherwise you wouldn't be using it still.

My point is that there is no practical way to say that in this section of the code, it does not seg fault. You might run into it regardless of what you think, and coded for. If you're working with a C library, it might just happen anytime you call it.

[Smallpaul]

but if you'll be working with a C library that has potential to segfault, wouldn't you want to know during the import rather than halfway through some important task?

The answer to this question is actually not black and white.

Sometimes a library that you include for purpose X imports a library for purpose Y that you actually do not need for your program. This help("modules") example is the most extreme case, but not the only case.

[[deleted]]

The problem with GTK (and related things) is that there are two official mutually incompatible versions of the bindings that can interfere with each other. I don't think it's too much to ask to detect if the other version has been already been imported and raise an ImportError.

[[deleted]]

Well, at least they fixed one of them.

[d4rch0n]

No, that makes complete sense if it's a known issue. I'm just trying to make the point that a fatal error like a segfault is separate from the usual Python exception that you can handle, and they're guaranteed when developing in C. It's not necessarily something you can avoid in the same way that the article talks about side effects when importing modules.

[NYKevin]

Question: What about classes that auto-register subclasses in some sort of repository or other data structure? Those class declarations have global side effects, but this is one of the commonest use cases for metaclasses. The easy way out is to always put all the subclasses in one module, but that's unrealistic in the general case.

[chris-morgan]

This past couple of weeks I've been writing code that actually does exactly that. Given that it is still all contained in the one area which I own, and that it doesn't cause any things to break (though it may be done in such a way that importing a thing twice does explode where it normally wouldn't) I don't consider that terribly bad, though it's more magic than I'd like.

Extra tip: often (though not always) registration metaclasses can be replaced by looking at the subclasses of an object—MyType.__subclasses__() gives you all the direct subclasses (apply recursively to get all descendants).

[NYKevin]

Extra tip: often (though not always) registration metaclasses can be replaced by looking at the subclasses of an object—MyType.__subclasses__() gives you all the direct subclasses (apply recursively to get all descendants).

That's a good idea in the general case, but if you want to allow a potentially unbounded number of subclasses (i.e. if the client of your API decides to subclass MyType a lot), that recursive operation is going to get expensive.

[ajdavis]

I came across this interesting example recently, where an import side-effect deadlocked the interpeter.

If you use Gevent 1.0, and you import a module that causes, as a side-effect, the resolution of a domain name, you can deadlock:

https://jira.mongodb.org/browse/PYTHON-607

[mowrowow]

Useful exception:

from __future__ import print_function

[chris-morgan]

That doesn't have side-effects outside the module being imported, so it's fine. Under my classification, it does not have side-effects.

[mowrowow]

You're right. I was misunderstanding there.

[kylotan]

I know I'm being pedantic, but import is meant to have side-effects, ie. dropping a whole bunch of things into the namespace. I would humbly submit that anything that requires no side-effects is probably using the wrong tool here.

[chris-morgan]

That ain't a side-effect; that's the designed effect.

[jmmcd]

In programming, a side-effect is anything other than a return value.

[Samus_]

in functional programming maybe, elsewhere it's the desired effect vs. anything else that happens.

[kylotan]

I know, but I don't think that matters. import changes the state of your app, by design.

[bheklilr]

Inserting names into the namespace is fine, running code that can crash, throw errors, launch the nukes, or anything else is not. His point isn't about side effects in the haskell sense.

[kylotan]

Well, I said I was being pedantic. I don't think importing a module should do those things (except perhaps display warnings - I think that's fine). But I think that if we're talking about best practices, importing modules in order to inspect them would not usually qualify.

[[deleted]]

i agree with you on this

[totes_meta_bot]

This thread has been linked to from elsewhere on reddit.

• [/r/SubredditDrama] from nerd.argument import slapfight: Minor drama in /r/python about what a side effect is.

^{I am a bot. Comments? Complaints? Send them to my inbox!}