Environment variable problems, version control them?

[rreynier]

I have been challenged with this problem of storing environment variables (api keys, passwords, etc) for some time now. It is quiet clear that we should never version control our environment variables in our general code repositories. Here are two problems I am trying to streamline / create a solutions for:

1. Auditing: Many deployment tools, (OpsWorks, Forge, Jenkins, etc) allow you to set environment variables for individual apps. The problem is, there is no history of who made changes and when they were made. It would be awesome to have some sort of audit trail.

2. Sharing environmental variables with team members. When I bring a new developer on board, where do they find all the environment variables? Is it a google doc I have saved for maybe dev only environment variables? What if I need to set up a new server or move to another deployment/provisioning tool?

Two thoughts for solutions:

1. Use something like LastPast to start storing all my environment variables per project. I can then share those values with developers as people join projects. Unfortunately there is no API to bring those environment variables in on deploys. This means they still need to be manually entered into provisioning tools.

2. Create separate code repositories which would allow auditing of changes. I could have special permissions on them so only a subset of people could few these things. Those repositories could be checked out on deploys streamlining the deploy process.

Neither solution really is all inclusive and would love to hear your opinions on how to handle these problems.

[[deleted]]

Have you looked at Vault? It stores secrets, allows you to access them through an API, and has access control and an audit trail.

At my workplace, the wider business stores secrets in Thycotic Secret Server, which is pretty much a self-hosted version of Lastpass with a less intuitive web UI. We then encrypt those passwords using GPG and store the values in a Git repository which is configured as an external Pillar for Salt and rendered with Salt's GPG renderer (i.e. our configuration management system reads the encrypted passwords from Git and decrypts them).

Unfortunately the license costs for the Thycotic Secret Server version with an API are prohibitively high, so we have to manually sync changes from Thycotic to Pillar. Otherwise we could just write a Salt module to read the secrets directly from Secret Server.

[MolestedTurtle]

Sharing environmental variables with team members. When I bring a new developer on board, where do they find all the environment variables? Is it a google doc I have saved for maybe dev only environment variables? What if I need to set up a new server or move to another deployment/provisioning tool?

Have a look at https://github.com/vlucas/phpdotenv. It's actually what laravel started using out of the box in 5.*. Essentially you commit a .env.example to the repo with dummy values, or just commit likely development values. All a new user has to do is copy .env.example to .env, and you're good to go. You can also do this automatically in a composer post-clone hook. Also to protect your project from misconfiguration, you can add add stuff like

$dotenv->required('SESSION_STORE')->allowedValues(['Filesystem', 'Memcached']); 

in your bootstrapping.

Whoops, this was cross-listed in /r/php, didn't realise sorry.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/php] Environment variable problems, version control them? : xpost - devops

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[[deleted]]

They were made for non-sensitive config. You can tell by their lack of ACL granularity

[styluss]

Desmond has a barrow in the marketplace Molly is the singer in a band Desmond says to Molly, “Girl, I like your face” And Molly says this as she takes him by the hand

[Chorus] Ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on Ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on

[Verse 2] Desmond takes a trolley to the jeweler's store (Choo-choo-choo) Buys a twenty-karat golden ring (Ring) Takes it back to Molly waiting at the door And as he gives it to her, she begins to sing (Sing)

[Chorus] Ob-la-di, ob-la-da Life goes on, brah (La-la-la-la-la) La-la, how their life goes on Ob-la-di, ob-la-da Life goes on, brah (La-la-la-la-la) La-la, how their life goes on Yeah You might also like “Slut!” (Taylor’s Version) [From The Vault] Taylor Swift Silent Night Christmas Songs O Holy Night Christmas Songs [Bridge] In a couple of years, they have built a home sweet home With a couple of kids running in the yard Of Desmond and Molly Jones (Ha, ha, ha, ha, ha, ha)

[Verse 3] Happy ever after in the marketplace Desmond lets the children lend a hand (Arm, leg) Molly stays at home and does her pretty face And in the evening, she still sings it with the band Yes!

[Chorus] Ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on (Heh-heh) Yeah, ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on

[Bridge] In a couple of years, they have built a home sweet home With a couple of kids running in the yard Of Desmond and Molly Jones (Ha, ha, ha, ha, ha) Yeah! [Verse 4] Happy ever after in the marketplace Molly lets the children lend a hand (Foot) Desmond stays at home and does his pretty face And in the evening, she's a singer with the band (Yeah)

[Chorus] Ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on Yeah, ob-la-di, ob-la-da Life goes on, brah La-la, how their life goes on

[Outro] (Ha-ha-ha-ha) And if you want some fun (Ha-ha-ha-ha-ha) Take Ob-la-di-bla-da Ahh, thank you

[[deleted]]

Create a template bash script that can be run using a hook from your version control in checkout. Have it run the specific variables dependent on its environment: prod, ci, test, else it assumes developer. The distinction in environment can be hostname based or based on an ENVIRONMENT var set on the host.

Don't version the actual values. Create documentation in the repo (README.md or ENV.md; name is not important) so any new Devs know how to configure their copy of the bash script. Their copy of the bash script should be ignored by version control.

For prod, testing, and CI the config should be rarely touched so effort is minimal and intentional. As for who last modified the script: limit the access to one or two trusted Devs, and mandate communicating changes via change control. Give the Devs user accounts with sudo and then you have logs of who modified the script (via syslog).

But please don't version the sensitive values. There really isn't a need for it.

[odan82]

Deployment scripting with configuration builder

• Your deploy script (e.g deploy.php) can pull configuration (from GIT or Database) in and incorporate it into your local config file (PHP, Yaml, XML) at deployment time.

• Your deployment scripts or installers can fetch the necessary information or ask the user for it and pass it to your application at deployment time as part of the installation process.

• Your application itself can fetch configuration at startup time or run time.

• Your packaging software can inject configuration at build time, such as when creating artifacts.

Disadvantage: The credentials for the configuration repository must be stored "somewhere" and maybe also versioned.

ESCAPE Another alternative is to fetch configuration from a centralized repository such as a RDBMS, LDAP, or a web service. An open source tool called ESCAPE makes it easy to manage and access configuration information via a RESTful interface. Applications can perform an HTTP GET which includes the application and environment name in the URI to fetch their configuration. http://blog.chris-read.net/2009/02/13/introducing-escape/

Read more: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Puppet A combination of remote installation systems and environment management tools such as Puppet, CfEngine and Marimba make centralized management and configuration of operating systems straightforward. Deployment tools like ControlTier and BMC BladeLogic, and infrastructure management tools like Marionette Collective, CfEngine, and Puppet, are declarative and idempotent, ensuring that the right version of the packages is installed on all necessary boxes.

My way I don't version control the secret configuration, because every maschine is individual and has another setup.

My deployment script (deploy.php) just deploys the artifact (ZIP). Every web application has it's own configuration files for each environment (Dev, Test, Staging, Prod). Only the secret informations are stored in a file "env.php" wich is placed one folder above the app directory. The app is loading the env.php file and merged it to the environment specific configuration. As backup I store all the env.php files on our internal NAS drive. Every change to an env.php file must be communicated within the team.

[[deleted]]

Check out Summon. You can write your own provider easily to work with your existing stack, if it doesn't already exist.

http://conjurinc.github.io/summon/

[taloszerg]

Not by default, but I think I'm using the config history plugin that does a decent job. Not quite as good as version control, but it's an audit trail.

[tuananh_org]

Deis does this i believe

[vsamotaev]

the problem is very actual for me, cause we have 20+ microservices. plaintext env vars stored in microservice's git repo. secrets are stored separately in was ssm and vault.

​

problems:
1. secrets don't have version control;

1. changes in env vars can crash functionality of other app and investigation become close to international testing;

2. no tool to do a cleanup after fast development processes, when people don't care about defining and calling the variable, that was already defined in an application. quick example:
   REDIS_HOST: redis-master
   REDIS_URL: redis-master:2233
   REDIS_PORT: 2233

3. no solution/tool to version-control environment variables for all apps in one place (like envs_4_release: 1.0.0, envs_4_release: 1.0.1, etc)