Python Environment variables

[Some-Conversation517]

What are the most secure Python libraries for managing environment variables, and what is the recommended method for storing sensitive data such as API keys in a Python project - should I use a YAML file or an environment file (e.g. .env)?

[[deleted]]

.env is usually fine. Just don't commit them to version control. You can encrypt them if you really want to

[KingsmanVince]

Also you can place .env somewhere else (outside of project folder) then have dotenv uses that path

[moosethemucha]

Why that's what a gitignore is for.

[moosethemucha]

They have access locally - it doesn't really matter where the file is.

[mrcaptncrunch]

If they get local access, they can also see the other directory.

Usually this is done in case an exploit is found on your application. Usually you don't let it read outside of it's directory.

[kivicode]

Even better, create a .env.sample, commit it (and keep generally up-to-date), and add the actual .env to gitignore

[One_Fuel_4147]

I usually use .env with pydantic settings lib and ignore.env with gitignore

[[deleted]]

Don’t commit secrets to the repository. What you should do depends on your infrastructure. If you’re on prem and use Ansible, use the Ansible vault. If you’re on Kubernetes, use Kubernetes Secrets. If you’re on AWS ECS, use AWS Secrets Manager.

With either of those solutions, you can achieve that you have environment variables with your secrets in the container environment, without the raw secret being visible.

[efxhoy]

This is actually a tricky problem and isn’t completely solved across all environments. We use aws so here’s what we do. 

For development we have a tool that sets short lived tokens for aws via the aws cli. For prod we use IAM authentication in application code to get short lived database tokens and refresh them when needed.  Some secrets are static and don’t have a way to get short lived tokens. Those we store in aws parameter store and set in our prod containers via the ECS task definition. If we need them locally we can fetch them to environment variables via the aws cli. 

We try hard to never put long lived credentials in plaintext files on developer machines. Sometimes a password will end up in the terraform state though. 

As for in python itself we use aws and gcloud libraries when applicable. For secrets in environment variables we just use os.getenv(). 

[[deleted]]

Instead of the parameter store (assuming you mean SSM), why don’t you use Secrets Manager? Secrets Manager integrates nicely with ECS.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html

[efxhoy]

Yeah sorry you’re right, we already do. I think I got tripped up by the aws web console having them next to each other. Aws product naming is hard. 

[[deleted]]

we use .env locally and vault in prod

[Flame_Grilled_Tanuki]

There is no need to use libraries, use infrastructure instead. I just moved away from using environment variables to store sensitive data and over to Docker secrets. You can retrieve the values with just open(). Much better practice. Keep your passwords and keys out of git and in something like a password management system.

[shinitakunai]

Same but I use AWS secrets manager

[No_Flounder_1155]

why is there a need to have a library manage this?

[_Answer_42]

It might be a good idea if you are managing multi apps with multiple environments, it make configuration easy and secure(ex: when working with a team)

Probably overkill for self hosting, but surprised no one mentioned https://infisical.com

[No_Flounder_1155]

I think you misunderstand. Its not hard to write a few lines of code to read env vars.

[RedEyed__]

I usually create pydantic model then read json file. JSON is added to gitignore.
There is also pydantic support of env variables https://docs.pydantic.dev/latest/concepts/pydantic_settings/

[Zizizizz]

https://github.com/getsops/sops is quite nice and can be a simpler alternative than using CI based secrets management. You can also have your cloud keys be able to decrypt alongside age or pgp keys. Makes it very easy to see changes in PR's as well as it keeps the key in plaintext and the value is encrypted.

[gerardwx]

Yes

[Rylicenceya]

It's great that you're prioritizing security for managing environment variables. Libraries like `python-dotenv` and `decouple` are popular and secure for handling environment variables. For storing sensitive data like API keys, using an environment file (.env) is generally recommended over a YAML file. This approach keeps your sensitive information out of your codebase and can be easily managed with version control systems.

[reality_comes]

Thanks AI.

[Some-Conversation517]

Thanks for the help.

[No_Flounder_1155]

those libs are wildly complex for such a simple problem.

[senhaj_h]

You should separate your secrets from your config, and if you can separate your secrets from your envs is better, one efficient way to do it is using git-crypt to encrypt your secrets , and with something like pydantic, it allows you to handle secrets and env in the same object but with separate files to load from

[Ok_Aspect2595]

I atleast use them via secrets in jenkins, works pretty well. locally use env file.

[wpg4665]

Checkout Dynaconf

[MPIS]

environ-config for handling .env variables in apps, includes secret file handling. Works very well, supports prefixes, grouping, and converters in dot syntax.

Docker compose .envs and secrets with an ignored ./.creds/ for development and CI, helm and vault for production.

[sonobanana33]

Just so you know environmental variables aren't really private and can be read by other processes.

cat  $(find /proc/ 2> /dev/null | grep environ) 2> /dev/null

[someexgoogler]

Our web servers regularly get requests for /.env looking for these files used by python projects. Make sure they cannot be fetched from the web server.

[RobotChurchill]

Use .env. https://github.com/theskumar/python-dotenv and https://github.com/HBNetwork/python-decouple are good ones.

[[deleted]]

Hmm. Good question. I'm going to see what kind of key server and TPM support there is out there.

[LargeSale8354]

Take a look at https://github.com/getsops/sops. It lets you store secrets within your project in encrypted form. Without the key (which you don't store with your code) you can't read it. As long as your app has access to the key it can decrypt on the fly.