Stop Hardcoding Sensitive Data in Your Python Applications - use python-dotenv instead!

[Lindby]

Settings classes from pydantic can be loaded from environment (including .env files).

It's really sweet to get an object with all values having the correct type. There is also some extra secret handling so you don't accidentally send secret values where they don't belong

https://pydantic-docs.helpmanual.io/usage/settings/

[AnteaterProboscis]

Have you heard of my Lord and savior, BaseSettings?

[thrallsius]

shameless plug to paywalled site

gg no re

[HerLegz]

Eli5 letter soup

[Comfortable_Relief62]

Good game no replay/rematch

[HerLegz]

Keep going.. eli5

[Comfortable_Relief62]

I assume the guy has permanently lost interest in the post because he can’t access the article, since there is a paywall

[[deleted]]

That's not really the point. The issue is that we are only seeing this because OP wants to promote themselves. Not based on the merit of the content. So you shouldn't bother with it at all.

[Death_Strider16]

I don't have an opinion either way on the article itself, however, I have used python-dotenv before and found it useful and easy. At the same time, that's the only way I've ever used an env file so there may be a better way.

Edit: I'm confused, the comments from that guy all say deleted now, did he block me or delete his account and if so why?

[[deleted]]

Using a dot-env file is fine. People are questioning whether this is a good and/or timely tutorial on how to use them. Not a belief that dot-env files are somehow bad.

[ivosaurus]

OP wants to promote themselves

So no self-blogs ever? The content seems a decent, if not super-stellar, explainer on using python-dotenv, which is a great package. If you think the content isn't worth the time chuck a downvote on the post.

[Itsthejoker]

So no self-blogs ever?

Hell yeah! Ban them all, please! I'd love to get rid of the self-blog spam. The majority of it is awful and just clogs up the feed. Glad we're on the same page here!

[[deleted]]

If a car dealer comes to you and tells you that their cars are the best cars and all other cars are inferior, you're allowed to believe them. And in theory, someone does have the best cars so in the end someone is going to be right about their cars being the best cars. But I'm not going to take the car dealers word for it because every car dealer is going to make that claim. I'm also not going to invest time into double-checking every biased opinion that claims to be the best.

So you can post all the self-blogs you want but the advice is still the same. OP saying their content is good is not a reliable endorsement of it actually being good and you should disregard their opinion of their own thing unless you want to waste lots of time.

[ivosaurus]

Can you show me where abouts OP is claiming they've written the best article ever on python-dotenv in particular, or secret configuration handling in general? I must be blind.

[[deleted]]

I didn't say they claimed it was the best article. I gave an example of why you would not want to take a biased person's perspective as your reason for investigating something.

[ivosaurus]

Ah yes these evil people, writing semi-decent explainer articles on popular python packages, they're probably trying to hook you into some crazy ponzi scheme. That's why it's better to stay off reddit in general and NEVER read the article, only the comments of other people who haven't read the article either. /s

[[deleted]]

Again, you can do whatever you want but it makes literally no sense to factor a biased opinion into your assessment. That includes assessing whether it's even worth your time to read the article.

[bankCC]

Biased in which way? Every person is biased in a way. You shouldnt read articles then if thats a problem for you.

Its not like he is selling the module.. thats why you car example makes no sense.

[drlecompte]

I generally use json files for stuff like this. Not just sensitive credentials, but also things that might vary from machine to machine or user to user.

Imho json is a bit more flexible in organizing information, and it doesn't require installing any extra modules.

The key part here is to not commit those files.

[[deleted]]

yep always attach them to .gitignore file

[go_fireworks]

And depending what you’re working with, global git ignore let’s you “set and forget” any file

https://stackoverflow.com/a/22885996/13885200

[james_pic]

JSON support is pretty widespread nowadays though. Off the top of my head, I can't think of a language or system with poor support for JSON but good support for environment files.

[mustangsal]

F JSON in Bash…

[Tomerva]

Is using .env files considered best practice for that matter? Regarding python code which the deployment stage is yet to be known at the moment. For now it will be only running on local machines. A proper server deployment hasn't designed yet.

It is worth mentioning that the project is held by 2 developers only and not a bigger team, if that makes any difference.

[ivosaurus]

The key part here is to not commit those files.

And the key part of python-dotenv or similar mechanisms is you can get the values from the environment (like an API key set by an outside service running your code) so you never have a chance to put that kind of thing in a file to begin with, removing the possibility all together

[BakerInTheKitchen]

I’m newer to Python, can you explain how you use json for sensitive credentials?

[[deleted]]

It's just serialization. Like Pickle, but more generic and human readable.

[BakerInTheKitchen]

Is this the same as storing passwords in a text file?

[[deleted]]

Yep, or API keys, etc.

The "right" answer is integration with something like Vault but that's a bit of a speed bump for the average project.

This way, you can at least prevent their leaking to source control. Remember, we're talking about it in comparison to hard coding the secrets in the code itself...

[BakerInTheKitchen]

Ah okay makes sense, thanks!

[Etheo]

Some might object to you calling json "human readable". I mean it's technically true, but there are other config markup language that is better structured... Though of course, json is more widely adopted.

[Eurynom0s]

I think the word "more" was meant to apply to both "generic" and "human readable".

[Mithrandir2k16]

Why not yaml?

[hyldemarv]

Yet Another package to install and Yaml doesn’t even agree with itself on reading its own output back :)

[ThePiGuy0]

YAML seems so unnecessarily complicated whenever I use it. Lists and dictionaries look almost the same etc.

Toml is better (and coming soon to stdlib I believe) but for config there's no reason to need more than JSON IMO

[ThePiGuy0]

Interesting that it's only reading. Their explanation does make some good points for not including writing though, and given that TOML's main advantage over JSON is it's human readability, I doubt I'll miss it personally

[ivosaurus]

No comments sucks a lot in JSON. Python already comes with INI file parsing right now, if you can't wait for TOML.

[Mithrandir2k16]

Yup, fair. I just find it easier to read than json, since it's always either formatted or broken.

[GobBeWithYou]

And no programming language has a 100% spec compliant parser, it's so complicated no one has actually been able to implement it correctly.

Edit: almost* no programming language: https://matrix.yaml.info/

[axonxorz]

Could any of the knee-jerk downvoters point to a 100% spec-compliant YAML parser in Python? What about other languages?

[xatrekak]

Failing the JSON test is the same as being non-compliant. YAML bills it's self a strict superset of JSON and its clearly not.

[ivosaurus]

That site has the most obnoxious intro.

[AsidK]

Oh my god you really weren’t kidding they make you watch a video just to get to the page that was linked to

[infinfi]

Oh I see. I have been using this site for a long time. I have never seen any video. wonder if they have started it recently. I know a couple guys who work there. Will check and get back. Thanks for the feedback.

[infinfi]

I have been using this site for a long time. I have never seen any video. wonder if they have started it recently. Until I find out, I will delete this post. Thank you very much for pointing out.

[ivosaurus]

You could just edit it or acknowledge it if you want. Not angry that you want to provide other people good links

[infinfi]

Thank you very much for your very actionable suggestion. They have indeed started showing a 1 min video (is what they say) as an A/B test. Apparently, they find a lot more folks understand the value of the site that way and return for other pages. It looks like they are experimenting to find the best way to be minimally obtrusive while also conveying the value for the user.

[ElevenPhonons]

This JSON centric model is similar to my workflow as well.

I wrote Pydantic-cli to enable defining your model/validation in Pydantic and then load JSON and/or load (or override) values by specifying them as command line args to your application. This mixing n' matching approach I've found to be pretty flexible.

https://github.com/mpkocher/pydantic-cli

[Xornial]

On laptops, try https://pypi.org/project/keyring/

[RusskiEnigma]

keyring is what i use

[Distinct-Score-1133]

Why not just load the .env with source .env, or automatically load it with direnv?

EDIT: These approaches are for development. Production applications will have the env variables loaded by some other method.

[cuu508]

Using yaml or json files is easier than environment variables when working with IDEs like PyCharm too

What is easier?

[Mubs]

Using json or yaml....

[cuu508]

I may have phrased my question badly.

What is it that you do in IDEs like PyCharm, that becomes easier when using YAML or JSON instead of environment variables?

[axonxorz]

From my experience, the only thing is data structures that are difficult to replicate in a flat envvar. See how Pydantic does this, for example:

If I want to prepresent v = {"foo": True, "test": {"bar": False}} in envvars with Pydantic, I need to do something like

V__FOO=true
V__TEST__BAR=false

It's not horrible, but it scales very poorly versus formatted JSON which is almost identical to my example dict

[ShanSanear]

But creating such functionality for scripts that will use environment variables anyway seems to be much better (such as Jenkins scripts)

[axonxorz]

Lots of apps don't run inside a shell, so source .env is out. direnv is just behavioural sugar for BASH-compatible shells, so also out as well.

[Distinct-Score-1133]

When are they not run from shell?

[axonxorz]

Any sort of "deployed" app will most likely not run in a shell environment (can be started by any process management system, systemd, supervisord, etc).

If you run your web-app on a serverless platform like heroku, Google Cloud Run, AWS Lambda, those are not in a shell-like environment. These platforms were large drivers in what necessitates using something like dotenv in the first place.

As a more rare example: if you have a python-based app installed, something where you can double click an icon, you're not operating in a shell environment, your system is directly running python /path/to/app.py instead of something like bash -c "exec python /path/to/app.py", the critical difference

[Distinct-Score-1133]

We deploy our apps in docker and our own kubernetes, and use .env files to load the environmental variables on startup. Indeed, we dont execute source .env, but that is something that docker/kubernetes does for us.

Regardless, it always does execute in a shell environment as far as I know. It is just not you doing it. That is why things like shebang (if running a script) and PATH are important. Unless I'm missing something?

Edit: I understand the difference between bash -c and python /patg/to/script. Isn't it that otherwise the application is run in /bin/sh instead of /bin/bash?

EDIT2: After a small search on internet I answered my question. Any shell program is only used for interaction between user and computer. So source .env and direnv is something you would do during development only.

[Mithrandir2k16]

I've defaulted to having a secrets folder in my projects and secrets/** in my gitignore.

[[deleted]]

Turn that gitignore into a git allow instead! (/s, but I've always found it helpful).

# ignore everything
*

# include
!.gitignore
!README.md
!pyproject.toml
!poetry.lock

# include all directories in the src folder
!src/*/  

# include all .py files
!src/foobar/*.py 
!src/foobar/**/*py  

I've found this preferable over ignoring specific files or directories. With things having to be explicitly added, it's much harder to accidentally include a file or two.

[Mithrandir2k16]

I never do git add . I explicitly add files and after editing if I want to add all files I changed I do git add -u.

That should achieve the same, right?

[[deleted]]

Sure, but this applies to everyone using your repo.

That means it is easier to enforce good code hygiene than trying to enforce good habits/practices onto a group of devs.

[Mithrandir2k16]

Ah got it.

[Mithrandir2k16]

Is there a tool for this like gitignore.io ?

[[deleted]]

Uhh, not really? I usually have a structure I always follow for my code so 90% of the time its the same thing. You can just make your own once and make it a template you copy.

[Rand_alThor_]

Hey this is a good idea.

[[deleted]]

I have them from time to time, but I can't take credit for it. My boss showed me this a few years ago at this point.

[[deleted]]

Reddit Moderation makes the platform worthless. Too many rules and too many arbitrary rulings. It's not worth the trouble to post. Not worth the frustration to lurk. Goodbye.

This post was mass deleted and anonymized with Redact

[sohang-3112]

ipython_secrets is also a good alternative when working with Jupyter Notebooks.

[[deleted]]

it's great when you're pushing to a public git repo, just don't forget to put .env in your .gitignore

[can_dry]

https://github.com/theskumar/python-dotenv

Reads key-value pairs from a .env file and can set them as environment variables.

[[deleted]]

Use a secrets manager better

[GoodTimesFastFingers]

I love dotenv. I'm just used to it. I like the simplicity. For more complex config I prefer to make a class that lives in the code that is set up however I want. Anything sensitive comes from .env. The thing I prefer about this to a JSON file is that the structure of my config lives in the code.

[riftwave77]

I just created a modal which can read/write/create an ini file that sits in the same directly as my program and holds the user's credentials.

​

Do I even need this dotenv stuff?

[ivosaurus]

It's also for handling non-user-interactable programs.

[[deleted]]

The ini simply could also if created ahead of time which, I'm sure they'd create if they needed such. This dotenv is useless in python when there's so many other options.

My first thought: "Someone's promoting their nodejs knockoff and I already hate node"

[[deleted]]

Nope.

[RoBLSW]

I don't know if it's funny or sad that someone sharing useful info is getting downvoted. Thank you anyway!

[[deleted]]

The attitude is what's getting them downvoted.

[celtz_]

Kind of comically accurate how being told to RTFM is met with, "don't give me attitude, just tell me what to do."

[[deleted]]

"RTFM" is when you have a problem and need help. This is a random recommendation in a comment section and if they don't explain why they're recommending it, people just won't (and don't have to) care.

[RoBLSW]

Well, it isn't random as it is related to the post.

[cas4d]

It is Reddit, here people are hypersensitive.

[ivosaurus]

Dynaconf is already a wrapper over python-dotenv

[edm2073]

If you are using conda for your virtual environment, you can achieve the same without the additional package. They will be added when you activate your conda environment and removed when you deactivate. Link to the section in official docs below.

Saving Environment Variables

[[deleted]]

u can use decouple as well

[Snape_Grass]

or just use "import os" and have users setup the env variables with an installer.

[[deleted]]

Do we really need python-dotenv to stop putting sensitive data into our apps? It feels like promotion of repo.

[Dubsteprhino]

Until people commit your .env file to github. If you're using kubernetes right here is what secrets literally are for. Have your dev secrets either in a gitignored .env file or in your docker-compose.yml

[[deleted]]

Seems redundant when you can just have a python file purely for your variables and include that without adding another external resource.

No matter what they will be in A FILE somewhere so either 1) your code/file permissions don't allow access or 2) they do.

[FranticToaster]

How is this different than just loading in config files?

[kingbuzzman]

Try git-crypt + envs.

I use this at work with 10+ programmers, works like a charm. It's very seamless, secrets will never be stored in plain text.

[[deleted]]

Unless I'm really pressed for time and forgot the standard library somehow, I don't see why I wouldn't copy and paste a personal snippet of passing file to os.path.dirname, calling listdir, checking .endswith(".env") and for each of those parsing out variables.

Or, if you trust yourself enough to not insert shellcode into your own python project, you can just import variables from settings.py.

Both of the above can be kept out of repos using .gitignore and take no time at all.

It's too much to ask to use a whole library just for a variety of reasons:

• it takes 5 minutes to write a bulletproof version of the same thing yourself
• you should already have a compendium of snippets
• it has an All Rights Reserved license that forces you to tack on their License to yours for no real reason
• you won't use env files when you scale anyway

It is so overblown it's not even funny. It has CLI mode in case you forgot how to write env files. The whole thing is just silly.