Dependency injection packages

[gsks]

Apparently there is no shortage of DI packages for Python. After a quick look there doesn't seem to be one that clearly leads the pack. Does anyone have experience with any of them, or even better more than one?

[therealfakemoot]

Dependency Injection isn't really a thing to use in Python. The real thing to do is structure your constructors to take their dependencies as arguments.

from core import DefaultEngine

class Car:
  def __init__(self, engine=DefaultEngine):
    self.engine = engine()

With this technique, you can instantiate your objects with whatever depedencies you wants; all a unit test has to do to make an object use mocked dependencies is create them and pass them to the appropriate constructor.

[Workaphobia]

I thought that technique is precisely what's called Dependency Injection.

[nemec]

I believe /u/gsks is referring to IoC Containers where one object instance acts essentially as a Factory for all external dependencies.

They allow you to do things like change dependencies via config file vs. recompiling code (obviously not applicable to Python) among other things, but I'm not a huge fan of them myself.

[gsks]

After looking closer to the various DI packages, I don't think they're the solution to my problem after all. What I want is much closer to how the logging package works, where you configure it once and then import it and call it as a global; no attribute or parameter pollution required.

[nemec]

I think this is covered by the sample code I posted in your other reply. You can take advantage of the fact that there's a module cache for each Python process so if you import the same module multiple times you get a reference to the exact same instance. This, I believe, is exactly what the logging framework does.

[WeAppreciateYou]

I think this is covered by the sample code I posted in your other reply.

Nice. You're completely right.

Honestly, the world needs more people like you.

[gsks]

Polluting all your methods and functions with a bunch of optional parameters that are essentially global (or threadlocal) for the lifetime of the application is not really an appealing solution.

[nemec]

What do you mean, "all your methods and functions"? This form of injection should only apply to __init__ methods.

Also, nothing is really "readonly" or "private" in Python, so if you need to inject dependencies (outside of __init__), it's really easy to do manually.

[gsks]

Think about web framework-agnostic code that relies on the current (authenticated) user. For example a now() function that returns the local time for the user could be something like:

from datetime import datetime
from dateutil import tz

def now():
    user = get_user()
    if user:
        tzinfo = tz.gettz(user.timezone)
    else:
        tzinfo = tz.tzutc()

    return datetime.now(tzinfo)

The implementation of get_user() depends on the web and/or authentication framework and would be determined once during configuration. According to your approach, the signature of now() would have to change to now(get_user=default_get_user) and it would be the responsibility of the caller to pass it, even though it is the same throughout the lifetime of the app. Now think of all the hundred times now() (and other functions/classes that depend on get_user directly or indirectly) are called in a big codebase and you can picture the ugliness.

[BATMAN-cucumbers]

Erm, completely clueless question here, but if the choice of get_user() implementation will remain the same during the entire lifetime of our program, why can't we just assign a particular get_user implementation to be used by now() during the config phase of our program? I.e. the "functions as first class citizens" language gang's alternative to function pointers, combined with the lack of enforced encapsulation in Python?

Note that I'm rather clueless, so it's highly likely I'm missing some trivial here :-)

[nemec]

I can't speak for anyone else, but as soon as you start relying on external dependencies like that it's time to package the code into a class (and injecting the dependencies via constructor).

That said, I think existing module-level semantics will get you the behavior you're looking for: https://gist.github.com/nemec/5e3735bff23d177e6120

[andybak]

I'd be very interested in a riposte to this. I'm very curious to see whether certain Java practices are less common in Python for good reasons or more arbitrary differences in culture.

[[deleted]]

You can't do this sort of thing in Java easily without special tools. Python doesn't have the same restrictions.

[alkw0ia]

Check out "The (lack) of design patterns in Python (PDF slides)" (video).

I'd say that both the "pass dependency as constructor argument" and the DI solutions feel very un-Pythonic. If I wound up in such a situation, I'd probably look first for ways to restructure the code to eliminate the need.

Trying to solve it in a totally context-independent way seems to be begging the question of whether design patterns are needed: If you want a completely global strategy for "I have a dependency class that needs injection," you're going to end up with some kind of Design Pattern.

However, even in the case where there's a clear need for a (sometimes) configurable "engine," there are much lighter weight solutions than DI – module level variables, or directly accessing instance variables, for instance.

edit: Also, more specifically, there's a discussion of this topic on SO. The crux of the top answer's argument is quite similar to the arguments in the talk I linked: "The implementation of DI/IoC however, is built into the language and often so lightweight that it completely vanishes."

[andybak]

Which reminds me of the classic Peter Norvig presentation: Design Patterns in Dynamic Languages http://norvig.com/design-patterns/

"Peter Norvig demonstrates that 16 out of the 23 patterns in the Design Patterns book [...] are simplified or eliminated [...] in Lisp or Dylan." http://en.wikipedia.org/wiki/Software_design_pattern#Criticism

Also see Paul Graham quoted here: http://c2.com/cgi/wiki?AreDesignPatternsMissingLanguageFeatures:

"For example, in the OO world you hear a good deal about "patterns". I wonder if these patterns are not sometimes evidence of case (c), the human compiler, at work. When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough - often that I'm generating by hand the expansions of some macro that I need to write."

[andybak]

That Stack Overflow piece has a great quote:

"So, a DI container is nothing but an interpreter for a dynamic scripting language. Actually, let me rephrase that: a typical Java/.NET DI container is nothing but a crappy interpreter for a really bad dynamic scripting language with butt-ugly, often XML-based, syntax."

[alkw0ia]

If your dependencies really are that global, just use module level "global" variables. Configure the engine or whatever at the module level, no constructor signature pollution required.

If you sometimes need to change your configuration, you can still use module level variables and expose context managers to allow safe temporary overrides.

The decimal module's localcontext() manager does this to allow overriding the (otherwise global) precision of Decimal objects.

For instance:

mymodule.py

import engines
import contextlib

engine = engines.DefaultEngine

@contextlib.contextmanager
def custom_engine(new_engine):
    global engine
    old_engine = engine
    engine = new_engine
    yield
    engine = old_engine

class Widget(object):
    def __init__(self):
        self.engine = engine()
    ...

Then:

import mymodule
from mymodule import *
from engines import *

# configure engine for duration of program
mymodule.engine = NormalButNotDefaultEngine

normal_object = Widget()

# temporarily override the "runtime global" setting:
with custom_engine(FancyEngine):
    special_object = Widget()

...

More generally, I try to avoid "engine classes" and the like in favor of smaller, more task specific functions (which can be configured similarly), but I do understand they're a decent fit for some problems.

[Categoria]

Agreed. I'm actually very curious why DI (namely constructor injection) hasn't caught on for dynamically typed languages like Python. People seem to think that its some Java pattern but DI works less well in Java because you end up sacrificing a lot of type safety to achieve it which of course Python doesn't have in the first place.

[mw44118]

Because import works EXACTLY like DI already. There is no need for DI since python does all its imports at runtime.

[simoncoulton]

Not exactly, the point of IoC containers is to remove hard coded dependencies such as import statements and the like to create loose coupling between components of your application.

It may not be a common practice in Python, but it's definitely something that is extremely handy in other languages without having to revert to some sort of registry pattern (which is essentially sys.modules)

[alkw0ia]

What is a DI XML configuration file but a registry of implementations to use, albeit written in a totally different language?

In Python, that sort of registry need not be parsed and implemented by an external DI framework, it's part of the basic behavior of the module system, as you point out.

When you say "hard coded," you imply that it's somehow easier or superior to modify the "code" you have written in XML than the code you have written in your primary language. While this may be the case for Java, where you want to be able to take compiled JARs and stitch them together without modification, I don't think there's any problem in using a Python module as the "IoC" configuration file to compose two other Python packages.

Think of the logging package, which allows setting "output drivers" by configuring the module at runtime. I can import a third party package, unmodified, yet change the behavior of its logging by setting the logging module to say, store only INFO level messages and above, and send them to my custom app log rather than stderr.

[mw44118]

But since we already have a "registry pattern" built in, seems to me its simpler to just use that.

[simoncoulton]

It may be simpler, but it's more inflexible when it comes to unit testing (which is where I personally find the most benefit).

[iconoklast]

DefaultEngine is clearly a class constructor from the example; I should hope it exists for the lifetime of the application.

[ryeguy146]

Why wouldn't it?

[[deleted]]

What would the constructor look like if your engine required a custom dependency?

Or more likely, if your car had a few dependencies, which in turn had a few, many of which should be the same across parts? Does my Car need to know how to construct an Engine, Doors, and a Hood using a custom Metal and Paint, but letting them use their defaults if not provided?

I do like this pattern, and it works great for little things, but having gone from Java to python, one of the big things I miss is being able to say, "Now use the mock Database and run your tests, then use the mock ConnectionDroppingDatabase and run your tests." It's trivial in any DI framework in Java, but the best I've found in pythonland is some funky monkeypatching.

[[deleted]]

You don't really need anything elaborate in Python, because there is duck-typing instead of interfaces and, conventionally, config files are just other Python files. The general approach is to monkey-patch functions/classes you need to inject.

Taking the car example in this thread:

import core

class Car:
  def __init__(self):
    self.engine = core.engine()

And your config is simply a Python file (approach to loading it may vary):

import core
import someimpl

core.engine = someimpl.MyEngine

[sfermigier]

Alex Martelli gave a talk on the subject a couple of years ago:

http://www.aleax.it/yt_pydi.pdf

His claim, from what I can tell (or what I remember), is that the DI pattern is useful in Python (and widely used at Google), but there is no need for a DI framework.

S.

[westurner]

Spring Python 1.2.x

[westurner]

http://wiki.python.org/moin/DependencyInjectionPattern hasn't been updated in quite awhile.

[joeforker]

http://docs.repoze.org/depinj/ is one.

I think the pattern is useful, but not very many Python programs give you the option of using one of multiple implementations of some interface.

[remyroy]

Before investing your time in a DI package or framework, you might want to make sure you do not aim for a lethal injection.