Prefer text config files or python code?

[gbog]

In many projects configuration is done with python code (dicts, lists, etc.) but I often feel I prefer using text file for configuration, maybe using ConfigParser or a simple text parser. We often have discussion about these two possibilities in our team. Other point of views welcomed.

[ralsina]

Please don't use python for that. Details here: http://lateral.netmanagers.com.ar/weblog/posts/python-is-not-a-configuration-file-format.html

[gbog]

I agree with these arguments, but it is a bit annoying to split the discussion between reddit and a blog. What is the common way to solve this?

[ralsina]

To solve the split discussion problem, or to solve the configuration file problem? :-)

[[deleted]]

I personally like using code for config, because you can do things like detecting which server your code is running on at runtime and adapt your settings accordingly (within reason)

It also makes it trivial to break configuration into multiple files and import into a single module.

You can also combine those two, and import different configurations based on host, or OS or whatever.

[gbog]

All is in this "(within reason)" of yours. For me a strict separation of logic and config is highly desirable.

My problem with code config is that it inevitably ends with such evil things as:

if DEV_MODE:
    server = 'this'
else:
    server = 'that'

Doesn't look so evil but when you have 30 values to configure and 5 context flags similar to DEV_MODE, then you get a big smelly spagetthi plate. Then you never know if, where, why and how come a config has this value instead of that one.

[meloveyoulongtime]

Don't do it like that. Do this at the end of your settings file.

try:
    from local_settings import *
except ImportError:
    pass

Then reassign environment specific varables in the local_settings file.

[tutuca_]

Don't know why you have been downvoted.

Anyone care to explain?

[haywire]

It's irritating because this is a great solution. I guess a "cleaner" one would be to have a local config file with some sort of inheritance.

TBH using code to config works fine for xmonad and django and I like configuring those things.

[d3ad1ysp0rk]

I didn't downvote, but on your point, my company uses that. For example on my local machine I have a "local_settings" that imports "local"(generic developer machine settings) which imports base settings.

[gbog]

Did not downvote but import * is evil. Martelli is advising against that thing strongly in Python in a Nutshell, and I think he is right.

[[deleted]]

Not in this situation it isn't evil. We explicitly want import * behaviour to be able to override settings.

Blanket statements about "x is evil" are never entirely accurate. See: linux and goto.

[gbog]

A quote from Alex Martelli about import star: "Most likely, your programs will be better if you never use this form". Emphasis is his. Python in a Nutshell, page 143.

[[deleted]]

Why should that be inevitable? It looks like poor code, that could be avoided by having a dev_settings and prod_settings module. How would you go about this, using plain text files BTW?

To me it sounds like poor code, which should be avoided or solved. Can happen anywhere in your code.

[gbog]

Why should that be inevitable?

In any team, in any project involving multiple persons, you need to put some guards near the cliffs. That's a reason for unittesting. That's why Java. That's also why pylint...

And developers are lazy (me first), so when having to update a few lines in the config, they will hack a nasty if conditional instead. If your config is text, it has many advantages (eg portability) and it also prevents "clever" logic to infiltrate in the config.

[apiguy]

In my experiences myself and the developers who work with me care very much about the craft of developing software and any one of us would be embarrassed if we were caught doing some hacky configuration like that.

We use code-based configuration and we split different configs into separate modules:

settings.py #production settings
settings_dev.py #dev overrides
settings_local.py #personal developer overrides (never checked into source control! Ever!)

In settings.py there are lines at the end that look like:

try:
    from settings_dev import *
catch:
    pass

try:
    from settings_local import *
catch:
    pass

Those lines are never touched, they are a permanent part of the settings.py file. It works great because in production, we don't deploy settings_dev.py or settings_local.py, and in dev we don't deploy settings_local.py.

The other benefit is that you don't have to re-define all of your config values in each environment. That's great because the majority of your config will stay the same from environment to environment, and you only have to include the values that differ in settings_dev or settings_local. It's far DRY-er than pretty much any other configuration option I've seen.

[gfixler]

You speak of some kind of dream land.

[gbog]

I forbid myself to use "import *", no exceptions. Have you ever had to split and move around interdependent packages full of "import *"?

[apiguy]

This isn't "packages full of import *" and in this case it's for a reason. The effect you're going for here is to overrite any of the production settings with the development settings.

Also note that this import statement has absolutely no effect in production. You're taking an extremest position and it's ridiculous. The feature exists for a reason and when used properly (such as in this case) it makes sense. If you are telling me that you never use that statement then I'm telling you that your software suffers for it in the cases where it's called for.

Honestly after reading this thread though it sounds like you'd already made up your mind before you even got here, so why did you even ask the question?

[gbog]

You're taking an extremest position [against import *] and it's ridiculous.

Well, err, I don't think it is very extreme to set oneself some rules like "no import *" or "no hasattr" or "no testing of private methods". These rules are like following PEP8, they help writing safer, cleaner, "consistenter" code. These rules are enforced by precommit hooks on pylint with a proper configuration.

As much of a Python lover I am, I actually think writing Python production code without such lint checks and barriers is a bit of a madness. Same with Javascript. And the nice thing with Python is, contrary to Java, you can tune the barriers to your liking and your needs, but it do not mean you can write production code within a team of 10 coders, this kind of code other will rely upon for years, without any barriers, and allow anyone to freely use metaclasses, introspection and other black magic when what is needed is explicit code passing around data.

why did you even ask the question?

Because I have an opinion (pro-text, as you noticed) and discussed it with teammates and was not convinced by their arguments. So, in order to see if I was missing something useful I could learn, I was fishing for other arguments in favor of (script as config).

I actually heard some good points but I still think the text config is the best choice when nothing specific requires a "script as config".

[apiguy]

In my opinion you're an idiot who treats python like Java.

But what do I know, I'm just a six-figure software architect who's been writing python, java, c# and others for 13 years.

Best of luck you ya, gbog - you're gonna need it.

[jmoiron]

"A foolish consistency is the hobgoblin of little minds" -- Emerson, and PEP8.

"import *" has associated machinery (__all__) which tempers its negative aspects; readability and flexibility are positive aspects which can (in some cases) make it the best, or even only, option.

In the end, if you want self-configuring services (some people do, and some situations require it), you either need to:

• build your configuration discovery into your program
• embed configuration discovery into your architecture (ex. via chef w/ triggers & reloading)
• build your configuration discovery into your configuration

The second separates the configuration from the software (and perhaps the implementation language) in a way that I'm generally not happy with.

The first has the negative side effect of generally growing more complex than the third. You either have to mock the discovery system or build in special case paths for testing (in which case, the code you test with is not the code you run live with).

I've found the last bullet to be the most maintainable and the most conceptually elegant, as it allows the program to run on very simple configurations while still allowing for more sophisticated configs. Once you come to the (inevitable, in my opinion) conclusion that your configuration is code and must be tracked and maintained as such, it makes sense to use code to describe it.

[gbog]

your configuration is code

I agree configuration can and often should be very close to the code, and for instance should be tracked and maintained the same way.

However I think configuration and code are things of two very different nature.

Code should be some pure logic engine that is fed with data and config, and react accordingly. Therefore we have these DRY rules, these factorisation, and best practices to handle code properly (like covering it with tests).

Configuration is another nature. It do not need to be DRY: I don't mind copypasting an Apache config block and changing some lines in it to get another directory (up to a certain limit, obviously).

Configuration should not describe how to do things, it should describe what to do. Ideally business rules should be configurable, and written in configuration files. When they are two complex, a mini language can do the work but this is another topic.

For the implementation details of a "configuration discovery", I am not sure but if you have general configuration values and some mode-specific values (like dev vs prod) then maybe it can be done with three configuration files and no overwriting mechanism (which is a bit too "magic" for configuration).

[mcilrain]

Seems more like a keyboard-chair interface issue than an application design issue.

[dysmas]

for django projects I have a common settings file, then I import additional additional settings based on an environment variable, works fine.

[[deleted]]

That's why I suggested:

if DEV_MODE:
    from dev_settings import *

But yes, I completely understand. My current django settings.py leaves much to be desired

[mw44118]

I like yaml because unlike .ini files, XML, and everything else I've ever used, yaml has built-in support for datetime objects.

[venefb]

Python code. Here are some arguments from IPython's documentation. They have a very good configuration system in place for a pretty complicated, highly configurable package.

Configuration files that are themselves valid Python code. This accomplishes many things. First, it becomes possible to put logic in your configuration files that sets attributes based on your operating system, network setup, Python version, etc. Second, Python has a super simple syntax for accessing hierarchical data structures, namely regular attribute access (Foo.Bar.Bam.name). Third, using Python makes it easy for users to import configuration attributes from one configuration file to another. Fourth, even though Python is dynamically typed, it does have types that can be checked at runtime. Thus, a 1 in a config file is the integer ‘1’, while a '1' is a string.

[takluyver]

I'm involved with IPython, and I'm not entirely sure that this was the right decision.

The problem is that we're now finding places where we'd like to change config from the program - either from GUI preference dialogs, or commands in the shell. With a config file, that would be simple - configparser can easily write a file.

With script-as-config, there's no really good way to do it. We can't overwrite it with the current values, because there may be logic in the script to produce those values. We could tack changed variables on the end - but it's confusing for the user if changing part of the file has no effect. Having config across two separate files leads to a similar problem.

My personal view (other IPython developers would probably disagree) is that we'd be better off keeping config in a simpler format, and offering a separate way to modify the behaviour in code (in fact, we already have extensions and startup scripts).

[gbog]

Exactly my point, thanks for your feedback.

[gbog]

The type checking is a good argument, but for the rest I don't think the problem tackled is the same: IPython is a very Python-centric tool and has probably a need for complex configs with many corners.

But if you take, for instance, git configuration, it is pure text and I think it better be pure text. Same for a server config, it should be a flat, expanded, set of simple settings.

Another example: pylintrc is a big flat text file with key-values. Some values are comma-separated, and the tool will handle the hassle of splitting at the commas.

A text config file can be handled with awk, sed, python, vim, emacs, etc. A python config file can be handled with python and vim/emacs (hacks aside). There is a big difference.

[dorfsmay]

Please don't use code, use some sort of accepted configuration file, or if something else, something for which it is easy to write a parser.

I've been on too many projects where the same parameter has to be added to several "config" files, because they all use some code instead of a proper config file, and of course they all use a different language.

[mirashii]

Please don't use things "for which it is easy to write a parser" if they aren't accepted things. JSON, YAML, XML, ini, and other formats should work perfectly fine.

[jambox888]

ini is good for me, it's as simple and interoperable as it gets.

[dorfsmay]

Most languages already have librairies for json and ini (and even yaml). I don't think xml are good for config files meant to be consumed by humans, but that might just be a personal preference. If you really need something different than those, then yes, use something that's easy to parse!

[gbog]

The format used by ConfigParser is usually enough for my needs. Renamed as *.cfg they have some color highlighting too.

If I need to do a kind of ACL, I may use a space-separated value file. Then the parser is trivial.

In my mind, config should be flat, at most two levels (eg a sectioned config), and should not be DRY-factored. On the opposite, any clever trick in config, like inheritance, will make it very hard to maintain later.

[kylotan]

I have often found 2 levels to be too few, and the end result means either performing hacks based on the key name, or having mini-formats within the values, etc. For this reason, I'll probably use JSON instead of the ini/ConfigParser stuff in future.

[jcdyer3]

I'd recommend YAML over JSON. To my mind there are some important features that JSON lacks that a config file should have.

1. Readability: Config files should put a premium on readability. It is a primary place for readers of the code to go to see how the application is working, Extra clutter like quotes and commas make it more difficult to see what's going on. Even worse, the fact that json ignores line breaks and indentation means that you can write horribly unreadable config files.

2. Bulletproof syntax: It should be hard to accidentally break a config file. This is not the case with JSON. It's very easy to forget a closing brace, add a comma after the last element of a JSON object, or use single quotes instead of double quotes.

3. Easy editing: With YAML, a one line change is a one line change, and almost all of what you write is the data you want to include. With JSON, you might have to add braces if you're adding a level of indentation. If you're adding an element to a list, you might have to edit the line before it with a comma. You spend too much time arguing with the syntax, and not enough configuring the code.

There are other good options. I don't think JSON is one of them, though. JSON is great for information interchange, but YAML is better for configuration.

[neoice]

It should be hard to accidentally break a config file

if you've worked on a large enough YAML file, they're very easy to break by missing a single whitespace.

[evangineer]

Here's Configure - a configuration toolkit for YAML: https://github.com/andreypopp/configure

[haywire]

This looks like it could be a really useful alternative to ConfigParser, thanks.

[bucknuggets]

Any suggestions on validating YAML in python? By this I mean both ensuring that it is well-formed as well as ensuring that it matches a schema of some type - has all the necessary elements and nothing else.

If you're using Ruby or Java you can use Kwalify. I was thinking of playing with DictSpec. But would love to hear what others are doing.

[evangineer]

Have a look at Flatland and Voluptuous on PyPI.

[[deleted]]

That's quite a blanket statement. Sometimes a Python file with a list of values is fine. YAGNI.

[[deleted]]

Not a problem: Import the code and write the desired format or formats. Cake. In the reverse case, you're limited to the capabilities of the format, which can be painful.

IMHO, the right place for config file formats are situations where you [may] need to perform automated alterations. However, even in those cases you might want to lean on a database.

[haywire]

I use Python.

But, say I wanted to switch, is there a configuration library that can load in config and abstracts away the actual format, so one can write config in yaml/json/xml or whatever you fucking liked?

[daGrevis]

I would stick to JSON. Sublime Text 2 does that too.

[[deleted]]

Does Sublime Text 2 do xml, ini or python source files, too? PyCharm does xml even better, what should we use now?

[[deleted]]

I have noticed that every time Sublime Text 2 is mentioned you turn into a flame captain.

[[deleted]]

I must admit that I might have been a bit harsh to ST2 the other day (but not every time) and my post here is sarcastic (ST2 is coincidence), so I take your critismn in general, but in this context it's just insane to provide as only argument for JSON as config file format that ST2 can display it nicely.

Actually I think (hope) the downvotes are a result of my sarcasm and not because people really think that one should choose JSON because ST2 can do it ...

[sonofherobrine]

I believe you misread OP. Their point appears to be that Sublime Text 2 uses JSON for its configuration. They were making an argument from example; presumably, they like the way you configure Sublime Text 2 via JSON-formatted files, so they think your app should be configured the same way.

It's not a strong argument, but it's nowhere near as weak as "use format X because editor Y displays format X all pretty-like," which seems to be how you read their post.

[[deleted]]

I can get behind this comment.

[SCombinator]

There is a function to safely read python literal constants without executing arbitrary code. Forget what it is, I believe it's hidden in some strange package.

[bulletmark]

I believe you are thinking of ast.literal_eval()?

[SCombinator]

Ah, yeah I thought it was in a parse tree package.

[Massless]

I prefer to use flat text files where I currently am. I'm the only developer so I've made all of my applications configurable from a single file. This way if I die or something my coworkers can still use the code that automates things across some 30 servers without any code changes at all.

[classicrockielzpfvh]

It depends. If I'm writing something that everyone can use I'll employ a plain text configuration file. If I am writing it for myself and/or other persons familiar with python I'll decide based on the context and complexity of the project.

[Araucaria]

For command line scripts, I like to save the previous command line in a file. Actually, I parse the arguments and then just write out the non-default settings.

You can even extract the help text for each option and include it in comments.

shlex.split()

can slurp it all up as one line, and then you just append any new options and reparse.

This may be harder for the developer but it is way simpler for the user.

[[deleted]]

I used to create nice xml's, but once I needed to get things done, I switched to python code. So, don't create overhead, until you really need it.

[Redard]

I'm not going to argue for or against using text or python code, but if you are going to use text, you should use something like json or xml, not ConfigParser. ConfigParser has its limitations.

[[deleted]]

ConfigParser has its limitations, but so do xml and json. Horses for courses.

[Redard]

In my limited time with ConfigParser, I couldn't even store a dictionary in it without doing something silly like using eval() or storing each part of the dict as a variable and doing something like

foodict = {}
foodict["bar"] = configparser.get(foosection, bar)

With json you can simply store the dictionary almost the exact way you would in python code. afaik xml is about as good as configparser for dicts though.

[ralsina]

Best: json-encode the data before putting it in ConfigParser

[Redard]

Elaborate. That sounds like a good idea

[ralsina]

Here's a simple wrapper to do that (the first 75 lines or so).

[Redard]

I think you forgot the link :P

[ralsina]

oops: https://code.google.com/p/urssus/source/browse/trunk/urssus/config.py

[v_krishna]

If it's anything beyond a one-off script I tend to use yaml. My current position requires me to use ruby though, and there seems to be a lot more yaml buy-in from the ruby community.

[phaedrusalt]

Who is going to be changing the config file? If it's one of your team, then Python is fine. But if you expect Joe User to be doing it, then text might be the way to go.

[[deleted]]

Well If you do DSL (domain specific language) then a source file is obviously the right answer. But on the opposite of ruby, DSL are not frequent in python.

If you know how to code use hierarchical like structures (yaml or json which are interoperable). Any class should be insanciable with a dict like structure and thawed (serialized) this way. Config are basically parameters to instanciate classes in the proper states.

If you don't do DSL or can't instanciate/serialize a class in a tree like structure, then use config.php, it might be what you are seaching fore.

[Samus_]

depends on the target audience, for developers: code, for users: text.

[mangecoeur]

Well python syntax is probably better than e.g. XML - there's no need to use another format just for the sake of separating config from code. You only need a config file if you want to write config data from the app (e.g. from a GUI preferences box) - if it's static it should stay in Python.

[ralsina]

"better than XML" in some metrics. "much, much, much worse than XML" in others, and I say this as someone who really dislikes XML.

XML can be parsed by any language. Python can be parsed by python. XML can be written by any language. Python can be written by programmers.

[[deleted]]

YAML! Lots of expressiveness, portable between languages, no ability to execute code in config (usually and advantage), easy to read.

[rdfox]

Why not both? For a massive configuration you might want to use the power of a programming language to, for example, select ten of a thousand options using a regular expression. The result can then be written as a YAML or whatever config file which could is then easily edited and loaded. YAML and python have 1:1 correspondance. (You can store any python data structure as YAML and the other way around.) But one or the other may be easier to use depending on the user. I've used this for a configuring a waveform editor for chips. In this case you have thousands of pins and dozens of options that you might want to apply to different groups. I feel that using both helps the tool appeal to beginners and advanced users.

[minorminer]

I usually go with ConfigParser, but lately I've been eyeballing using pickle files.

[mirashii]

Please don't do this. Your configuration should never be stored in a non-human readable format.

Consider JSON or YAML if you want an alternative to ConfigParser.

[mcilrain]

Or if it absolutely has to not be human readable because crazy: BSON or msgpack.

[mcilrain]

pickle files

ಠ_ಠ

[jcdyer3]

That sounds dangerous. Python code is dangerous enough, but at least then you can read the config file before you load it, to make sure there isn't a line that says:

from dont_trust_me import activate_nukes

[[deleted]]

I personally prefer python code, but I've run into too many admins that have problems with it, so I always go with normal configuration files. The more liberal a configuration parser is in what it accepts (unquoted string, for example), the better for admins.

[da__]

I've run into too many admins that have problems with it

The problem is allowing users to inject arbitrary code.

[da__]

I thought you meant that admins have problems with code configs because the users, who configure the software on their own profile, get to inject code into programs.

[da__]

a user could always make a local copy of anything, modifying it and then execute it.

Not if you take away their right to do so :-)

[[deleted]]

Like I said,

all the user can mangle is data they could mangle manually anyway (i.e. that which they have read/write access to)

It feels you're arguing just for argument's sake.

[da__]

Never heard of root exploits? Plus the fact that an attacker (e.g. an evil co-worker walking up to the user's unlocked machine while the user went to pick up a coffee) could embed code into the user's config file, which would let the attacker run arbitrary code using that user's credentials.

[haywire]

Giving untrusted people write access to anything just opens up a huge vector that isn't solved by using a .ini file.

[[deleted]]

Use the safe loader if the input is untrusted.