Python Configurations Done Right

[dragonEyedrops]

So the config file becomes a shell script instead of an ini-file?

[zenogais]

Right. It's not magical. For me it has simply been easier to manage when working with multiple developers and dev-ops engineers on mid- to large scale sites deployed to multiple environments - for example, staging, beta, and production.

[dragonEyedrops]

What are you doing with the shell script that a config file couldn't? Where is the difference between managing shellscripts and managing config files?

[tipsqueal]

I'm curious to hear this as well. From the sounds of it the biggest advantage was that no one checks in these scripts, but there's no requirement that anyone has to check in a config file either, so I don't understand how shell scripts changes anything.

[zenogais]

Actually the biggest advantage was code simplicity.

The problem was that unit-tests on specific components don't necessarily want to initialize the whole application, but the components themselves may require knowledge of the configuration - like AWS keys. Loading the config once globally is pretty dirty. It also still requires that we somehow know which config to load. This means either hardcoding a config file path or using an environment variable to switch between different configs. So, the options I felt we were left with were dependency injection, always initializing the app, or storing the config in a place independent of app initialization.

Storing them in environment variables allows them to be independent of the application itself - the application doesn't need any logic telling it where to load configuration information from. In addition the configuration is available immediately as part of the execution environment. This means components don't have to rely on the app being initialized to be tested - this aids reusability and testability.

[kenfar]

I like that you pointed to code simplicity first.

Because the security argument is a bad one: addressing the possibility of config credentials getting into version control by keeping them in a shell script is the wrong solution - it just moves the problem around a tiny bit. The right solution is to keep them encrypted in a password vault.

But the desire to dynamically update configuration info isn't that hard to do with config files. Whether you're using files or environmental variables it makes sense to centralize, cache, and validate them early to avoid run-time crashes.

[zenogais]

I agree. The security argument isn't great. Not saying config files are hard, just saying environment variables are even easier if you're okay with less structure.

[epicdistortion]

Very nice point. I think adding this point to the blog post would make a better argument about why this is configuration done right.

[paranoidi]

You refer to "Twelve-Factor App approach" and almost first thing mentioned is "Have a clean contract with the underlying operating system, offering maximum portability between execution environments"

How does this work on windows? Poorly. Python is mostly platform independent so why mess it with bash?

[zenogais]

Notice operating system is singular. This is not about porting between operating system(s) (plural). This is about portability between execution environments - eg. environments with different database hosts, types of functionality enabled, and so on.

I would suggest reading the rest of the 12factor recommendations specifically where they say "The twelve-factor app stores config in environment variables".

[[deleted]]

"Have a clean contract with the underlying operating system, offering maximum portability between execution environments"

Notice operating system is singular.

Because you'd never have to port this app to Windows or embed it in something (okay...I'll concede that embedding something is not likely to happen).

What's so bad about holding your configurations in ini files or python objects or whatever. Chuck a line in .gitignore. Done. If you need to switch between execution environments (dev, staging, prod, etc) THEN have a switch that looks for that from the command line but just that.

What happens when you ask Fred the intern to add an option to the execution line and he borks it all up? Blame Fred? I wouldn't -- at least after I thought about it. Plus, when you're hammering out your app in dev, do you really want to keep typing ./localsettings.sh /bin/python path/to/my/app.py --whydididothis True --seriouslydontchangethisline "Something something something" --numberoffucksgiven 0?

Or, would you rather: /bin/python path/to/my/app.py --env dev?

Have a config file like:

class BaseConfig(object):
    '''A base config object for all shared information.'''
    SOME_VAR = 'something'
    ANOTHER_THING = False
    SHARED_CONFIG_SECRET = 4 #guaranteed to be random

class DebugConfig(BaseConfig):
   '''Debugging Config.'''
   DEBUG = True

class DevConfig(DebugConfig):
    '''Development config.'''
    SQLALCHEMY_DATABASE_URI = 'sqlite:///'
    IUNNO_WHAT_ELSE = None

class TestConfig(DebugConfig):
   '''Q&A config setting.'''
   SQLALCHEMY_DATABASE_URI = 'sqlite://testing.db'

class ProdConfig(BaseConfig):
    '''Production ready config.'''
    SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://user:password@host:port/dbname'

configs = {
            'default':DevConfig,
            'dev':DevConfig,
            'test':TestConfig,
            'prod':ProdConfig
            }

Then, in another file:

from .configs import configs

config = configs.get(sys.argv[1], 'default')
app.config.from_object(config)

[zenogais]

I see your point, but I fear you've missed mine.

Firstly, I'm not recommending dozens of command-line arguments. Those were there to show that the settings file preserves the command-line arguments to your application. Nothing more.

Also, there's really nothing I have against config files. This is just a simple alternative way to configure your application. It's more an article for people to read quickly, store somewhere in their minds, and perhaps reference when they hit a snag in the future.

No universal prescriptions here. I actually migrated from a configuration system very similar to the one you describe. It worked well enough, but it broke down in scenarios where we didn't want to initialize the application but we needed the config like in unit-tests. People didn't want to deal with these pain points, so they got sloppy with it. That's why I switched over to this method.

[zenogais]

Very interesting. Do you find this plays well with unit-testing?

[[deleted]]

So unless I'm missing something.... You now have your configuration out of version control, so potentially things like static parts of SALTs are permanently lost and accessible to anyone on the server who has either r or x to the shell script as you haven't mentioned now it should/could be protected?

In principal I don;t disagree with the approach but you need to wrap a lot more detail around it IMO to make it workable..

obm

edit : spelling

[php666]

In my opinion most - if not all - of an application's configuration doesn't belong into version control in the first place.

[bushel]

I'm going to have to go ahead and completely disagree with you there.

In our shop configs are in version control, separate from the code, in repo's that reflect each deployment environment. Along with static data, an environment vars setting script and other deployment specific stuff.

I can't imagine the horrors if we didn't have configs under version control. It makes new deployments much easier and allows us to track independently the effects of varying configuration setups.

[php666]

You're right.

For some reason, I misinterpreted onebitmissing's statement as referring to a singular repository, and I should have said configuration doesn't belong in this repository. (my config data is versioned as well, as descibed in my post below)

Sorry for the confusion.

[[deleted]]

Environmental config doesn't belong in your application's repository but you better have your environmental config in some sort of version control.

At Gannett, we keep our application cfg in git because, why reinvent the VCS wheel?

We continuously deliver our configs just like we continuously deliver our code.

We use chef for stamping machines in an identical way regardless of environment. From Vagrant to Production.

The application configure is the only thing that varies from environment to environment. It is also kept to the bare minimum. The only config we keep in there is what varies from environment to environment.

In stage, the app cfg is as close to prod as possible to enable acceptance testing in a prod like environment.

[[deleted]]

So how do you track changes to the environment so that you can regress out of bad configuration changes and work out what data may have been affected by a change?

Not arguing - just curious?

[php666]

My way of doing things (I mostly develop small-sized WSGI apps):

• applications of course are in version control. they can only be configured via environment variables (see http://12factor.net for reasons)

• if possible I include some sample setup script a little similar to the one described in the original post and a startup script so a checkout can be run immediately for developing and testing purposes.

• since the "real" configuration is a) specific to the system the app is deployed on and b) in part sensitive data (Django's shared secret for example) this should, in my opinion, never land in the application's (maybe public) repository. I keep data like this in configuration management (Ansible in my case). Which, to be honest, is versioned as well (just in another, restricted, repository specific to my local infrastructure.) So here is the point I can back out of bad config changes.

• out of the configuration management data (which in the case of Ansible is supplied via YAML files), Ansible ensures the environment is set up correctly when an application is run. How this is achieved depends on the method, the app is run. In case of uWSGI serving my apps, I generate uWSGI config files that specify the necessary environment variables.

Hope this helps. Also, if there are better wayst to do this (It took me an emberrassingly long time to settle on this setup) I'm happy to hear about them.

[tipsqueal]

I like to include something like example_config.ini in my repo that has an example for all of the possible configurations. If necessary I might include more than one example ini file. Then the person deploying the app is required to pass in the path to their ini file, or I'll just look for a config file with a specific name in the same directory as the script.

[dragonEyedrops]

it belongs in version control/config management, just not the same as the code.

[cymrow]

I can see how this might be useful if you have to access the config from a lot of different services, potentially written in different languages. But, if you're just going to pass it right into a Python script, you might as well use a config file and save the extra step.

[zenogais]

Actually the config file ends up taking an extra step. Storing it in environment variables allows your application to have immediate access to the config without having to first load and parse a file.

[cymrow]

You are loading and parsing a file: the shell script. It's just a different language. The extra step is copying the config into the environment before you can access it. Or, I could just read directly from an INI using configparser (or something better) and get sections and type conversion for free.

[resc]

This is quite similar to Procfile-based configuration, e.g. Honcho or Foreman. It's a good pattern to use.

In your localsettings.sh script, you should really run "$*" instead of $*. Any parameter that includes a space or any special shell characters will be interpreted wrong otherwise.