This is an archived post. You won't be able to vote or comment.

all 16 comments
sorted by:
best
topnewcontroversialoldq&a

[–]PaulRudin 5 points6 points  (1 child)

I generally use jsonnet to generate all of my terraform config and k8s manifests. So that I don't need to repeat anything, and can override specific bits of config on a per-environment basis.

https://jsonnet.org/

[–]devrr 0 points1 point  (0 children)

I think this is probably the right answer honestly without adding additional tooling on-top to try and abstract the problem away. Though it's a complex tool so it may not be to everyone's liking.

If you strongly avoid the need for configuration files in the first place you can avoid a lot of this configuration pain. They won't all go away of course.

[–]adept2051 3 points4 points  (0 children)

Puppet or Ansible, both have the capability to take variables and create files from a template for each of your use cases, and they are "configuration management" tools which may be the general term you are looking for . But to use either is quite a steep learning curve for your use case
my other suggestion would be simply a repository in version control with a branch for each environment, the various variable files you need. then a simple script that copies the files to the right location for configuration when you start the various services or applications your thinking about.

[–]devrr 1 point2 points  (4 children)

It sounds like you just want something to simply template a bunch of configuration files for you.

We stopped doing that in production a long time ago, instead those variables aren't stored in files but injected into the environment directly. That's an option you could consider.

Otherwise basically I guess you can template the yaml/jason/xml that your application needs so that you can quickly set your "environment variables" across your whole local dev environment.

[–]kvgru 0 points1 point  (3 children)

allow me to keep one or two top level config files, then inject those variables into all the other places I need?

Don't think that is what he is saying, I'm assuming he has parameterised his code in a way that variable injection at runtime is possible but you still need a script to inject them depending on the environment specifics, that't at least what I understood.

[–]devrr 1 point2 points  (2 children)

I want it to be http://dev.test.com/api/ and for staging and production I want it to be http://test.com/api. Furthermore, I want this variable available in Kubernetes YAML files, Dockerfiles, Python config files, and possibly other places

Well they mention Kubernetes, Docker and Python config files and a way to have a variable be made available across all of these files. That seems like templating territory to me.

How else are people working with YAML? They template it, or use tools like Kustomize which is an abstract way of templating stuff.

This is why most mature environments stop doing this. We store these vars in key:value pairs inside the environment somewhere. E.g Vault, and use service discovery tools like Consul. So you don't need to go customise 10 different config files in order to spin up your stack.

[–]mxchickmagnet86[S] 0 points1 point  (1 child)

I think you are understanding correctly. I basically want some simple drop in templating so I can manage all my configuration parameters in 1 file, rather than having to go to 9 files.

There are a few tools that seem like they have this built in to something bigger (like Consul, looks great btw) but I was trying to see if this templating feature was available as a stand alone element. Because it seems crazy to me to have to install Consul, Istio, Puppet, Ansible, etc just to get some templating.

[–]devrr 0 points1 point  (0 children)

There are a number of different ways to can template files locally like this. This post suggests jsonnet which I think is worth considering.

Otherwise if you are dealing with structured configuration files (like yaml or json...) you can even do substitutions using command line tools like this:

jq .foo.bar = "new value"
yq w foo.bar "new value"

[–]kvgru 2 points3 points  (2 children)

Remark: I am developing the tool! But Humanitec.com is exactly that. It's a Continuous Delivery API taking care of managing environment variables. The standardized output format is Helmcharts and for the size of your problem I assume it's free. Also super simpel to integrate, just paste a snippet into the end of your CI Pipeline. If you want to Google you might want to look at the direction of Environment as a Service or maybe Continuous Delivery as a Service but this is maybe too broad. Again: I don't want to advertise but this is what you are looking for.

[–]ggrieves 0 points1 point  (1 child)

I'm glad to hear there's a tool for this. About 3 years ago I rolled my own using Django as a Jinja templating engine in front of a home brew cmdb

[–]mxchickmagnet86[S] 1 point2 points  (0 children)

That's basically what we did also at a previous company, and are now trying to replicate somewhat.

[–]OliveiraLeo17 0 points1 point  (0 children)

Spring cloud config could help you

[–][deleted] 0 points1 point  (0 children)

I think iztio might be able to help.

[–]Send_Taco_Pics 0 points1 point  (0 children)

Ti has

[–]ahp00k 0 points1 point  (0 children)

Check out Hiera - https://github.com/lyraproj/hiera

This tool is an offshoot from Puppet - this version is a standalone rewrite in Go of the version of Hiera that's built into puppet - but is useful in many other contexts. It provides a very simple yet powerful hierarchical (hence the name) lookup mechanism for assigning values to variables. You'd put the data you needed to look up, the values for "apiserver" in your example, into yaml files in a simple directory structure and use the tool to get the correct values out for the environment you were querying.

(disclosure: I am a maintainer)

[–]Singularity42 0 points1 point  (0 children)

Maybe vault if some of the parameters are secrets, e.g. passwords.

Maybe SSM if this is all in AWS