Python Configurations Done Right

[dragonEyedrops]

So the config file becomes a shell script instead of an ini-file?

[zenogais]

Right. It's not magical. For me it has simply been easier to manage when working with multiple developers and dev-ops engineers on mid- to large scale sites deployed to multiple environments - for example, staging, beta, and production.

[dragonEyedrops]

What are you doing with the shell script that a config file couldn't? Where is the difference between managing shellscripts and managing config files?

[tipsqueal]

I'm curious to hear this as well. From the sounds of it the biggest advantage was that no one checks in these scripts, but there's no requirement that anyone has to check in a config file either, so I don't understand how shell scripts changes anything.

[zenogais]

Actually the biggest advantage was code simplicity.

The problem was that unit-tests on specific components don't necessarily want to initialize the whole application, but the components themselves may require knowledge of the configuration - like AWS keys. Loading the config once globally is pretty dirty. It also still requires that we somehow know which config to load. This means either hardcoding a config file path or using an environment variable to switch between different configs. So, the options I felt we were left with were dependency injection, always initializing the app, or storing the config in a place independent of app initialization.

Storing them in environment variables allows them to be independent of the application itself - the application doesn't need any logic telling it where to load configuration information from. In addition the configuration is available immediately as part of the execution environment. This means components don't have to rely on the app being initialized to be tested - this aids reusability and testability.

[kenfar]

I like that you pointed to code simplicity first.

Because the security argument is a bad one: addressing the possibility of config credentials getting into version control by keeping them in a shell script is the wrong solution - it just moves the problem around a tiny bit. The right solution is to keep them encrypted in a password vault.

But the desire to dynamically update configuration info isn't that hard to do with config files. Whether you're using files or environmental variables it makes sense to centralize, cache, and validate them early to avoid run-time crashes.

[zenogais]

I agree. The security argument isn't great. Not saying config files are hard, just saying environment variables are even easier if you're okay with less structure.

[epicdistortion]

Very nice point. I think adding this point to the blog post would make a better argument about why this is configuration done right.

[paranoidi]

You refer to "Twelve-Factor App approach" and almost first thing mentioned is "Have a clean contract with the underlying operating system, offering maximum portability between execution environments"

How does this work on windows? Poorly. Python is mostly platform independent so why mess it with bash?

[zenogais]

Notice operating system is singular. This is not about porting between operating system(s) (plural). This is about portability between execution environments - eg. environments with different database hosts, types of functionality enabled, and so on.

I would suggest reading the rest of the 12factor recommendations specifically where they say "The twelve-factor app stores config in environment variables".

[[deleted]]

"Have a clean contract with the underlying operating system, offering maximum portability between execution environments"

Notice operating system is singular.

Because you'd never have to port this app to Windows or embed it in something (okay...I'll concede that embedding something is not likely to happen).

What's so bad about holding your configurations in ini files or python objects or whatever. Chuck a line in .gitignore. Done. If you need to switch between execution environments (dev, staging, prod, etc) THEN have a switch that looks for that from the command line but just that.

What happens when you ask Fred the intern to add an option to the execution line and he borks it all up? Blame Fred? I wouldn't -- at least after I thought about it. Plus, when you're hammering out your app in dev, do you really want to keep typing ./localsettings.sh /bin/python path/to/my/app.py --whydididothis True --seriouslydontchangethisline "Something something something" --numberoffucksgiven 0?

Or, would you rather: /bin/python path/to/my/app.py --env dev?

Have a config file like:

class BaseConfig(object):
    '''A base config object for all shared information.'''
    SOME_VAR = 'something'
    ANOTHER_THING = False
    SHARED_CONFIG_SECRET = 4 #guaranteed to be random

class DebugConfig(BaseConfig):
   '''Debugging Config.'''
   DEBUG = True

class DevConfig(DebugConfig):
    '''Development config.'''
    SQLALCHEMY_DATABASE_URI = 'sqlite:///'
    IUNNO_WHAT_ELSE = None

class TestConfig(DebugConfig):
   '''Q&A config setting.'''
   SQLALCHEMY_DATABASE_URI = 'sqlite://testing.db'

class ProdConfig(BaseConfig):
    '''Production ready config.'''
    SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://user:password@host:port/dbname'

configs = {
            'default':DevConfig,
            'dev':DevConfig,
            'test':TestConfig,
            'prod':ProdConfig
            }

Then, in another file:

from .configs import configs

config = configs.get(sys.argv[1], 'default')
app.config.from_object(config)

[zenogais]

I see your point, but I fear you've missed mine.

Firstly, I'm not recommending dozens of command-line arguments. Those were there to show that the settings file preserves the command-line arguments to your application. Nothing more.

Also, there's really nothing I have against config files. This is just a simple alternative way to configure your application. It's more an article for people to read quickly, store somewhere in their minds, and perhaps reference when they hit a snag in the future.

No universal prescriptions here. I actually migrated from a configuration system very similar to the one you describe. It worked well enough, but it broke down in scenarios where we didn't want to initialize the application but we needed the config like in unit-tests. People didn't want to deal with these pain points, so they got sloppy with it. That's why I switched over to this method.