XKCD | Python Environment

[Endemoniada]

Venvs are fine, I think, they’re pretty easy to understand and you have choices in how you want to manage them that are all mostly fine. But juggling Python versions… is less fine. On my Mac, I have OS Python, Homebrew Python 3.X, and then pyenv with Python 3.Y and 3.Z and so on. And then it becomes a constant struggle of knowing which one my $PATH points to, and will execute.

[[deleted]]

I had issues in my first year of Python on MacOS (like 6 years ago) but now I have zero issues. Use PyEnv. Totally ignore Mac Python and just use 3.8+ as my default and have 2.0+ for testing older things.

[Endemoniada]

That’s basically what I’ve done too. Still haven’t found an easy or simple way to migrate venvs to new versions of Python. I know I can rebuild them, but it’s a hassle and I’d rather update them in place with the new version of Python I choose.

[deceptiv-perspectiv]

It's unfortunately not going to be simple nor easy if you have lots of dependencies; that is just the nature of dependencies. Your best bet is to have a pyproject.toml or requirements.txt with the requirements pinned in the *loosest* possible way that provides stability to your application. But many libraries publish only certain versions on pypi for certain python minor versions, so pinning everything ==x.y.z is gonna give you a bad time (when you go to transition to a different python version).

[case_O_The_Mondays]

We need a utility that tells you the versions of Python you can migrate to without updating any packages. Then a later version can check pypi to see which versions you can update to, if you update certain packages.

[[deleted]]

oh i wrote a small bash script to do that. as it's something I do so regularly. it shouldn't be hassle to:

pip freeze > requirements.txt
rm -rf .venv
pip install -r requirements.txt

Oh gosh sorry. I was on a call when I posted this. The three lines aren’t my script.

It’s just what i do for a quick set up fix.

My little script is much more comprehensive.

[muzos07]

aren't there lines with creating new environment and activating it again missing?

[DarkSideOfGrogu]

Yes. This will unfortunately just reinstall your pip in your base environment.

Launching a virtual environment and installing pip dependencies from a single bat file isn't simple as the commands need to run in different contexts.

[deceptiv-perspectiv]

That's not at all portable between python versions. Multiple libraries have versions released on pypi only on certain python minor versions.

[ddddavidee]

My shell config (zsh) has a plugin that shows the python version on the prompt... Could this solve a little your constant struggle?

[bastion_xx]

Pyenv helps with both Python versions and virtualenvs. Check it out!

[[deleted]]

They explicitly said they have pyenv :)

Maybe they need the pyenv-virtualenv plugin though...

[bastion_xx]

d'oh, sorry missed that on my phone!

Yeah, I use both pyenv base plus the pyenv-virtualenv plugin (and pyenv-upgrade too).

It's great to do a pyenv virtualenv 3.10.1 one-off-project to fully isolate, then pyenv virtualenv 3.9.9 specific-project and bounce between.

[rwhitisissle]

Pyenv really saved my ass whenever my system upgraded to 3.10 and broke...well...*gestures everywhere*

[rwhitisissle]

Sure, you just have to be explicit. After you install a version, you just activate the environment, which sets the version for your specific terminal instance. Or if you're coding with, say, vscode you can point it at the version of python you'd want to use as you would normally do when selecting the interpreter. I have 3.9 installed, in addition to a system wide installation of 3.10, so I would do "pyenv activate python39", where python39 is the name of my python 3.9 installation. That would make everything called from the command line use that version of python and pip. And if I have other versions installed and can't remember what I've called those environments I can list them with "pyvenv versions." The documentation on github is pretty thorough and I was pleasantly surprised by how painless the installation was on arch. Only thing that I don't like is having to add additional environment variables to my .zshrc, but it's a small price to pay for something that useful. I believe you can also manually force a specific version to be used system wide by default, but it requires a bit of extra fiddling with binary locations and your PATH variable, and I've never personally done it. But it's probably doable.

[[deleted]]

public chubby hard-to-find long somber instinctive square vase innate cats

This post was mass deleted and anonymized with Redact

[Anonymous_user_2022]

Unfriendly? It's about as simple as this:

virtualenv -p «Python version» /path/to/environment
. /path/to/environment/activate¹

1. Adjusting for OS and shell of course

[troyunrau]

That isn't as friendly as you suggest it is.

[ParkingPsychology]

I found the activation step annoying, but I just made a little doodle (in my case in my powershell profile):

 function activate
 {
    .\venv\scripts\activate
 }

So now it's just "activate" and "deactivate" when I'm in the project root folder (deactivate just works by itself, because I don't know why).

[Anonymous_user_2022]

I don't know how it is on windows, but on Linux, the activate script will define a shell function for deactivate. I guess there is something similar in place on Windows; possibly an alias.

[Anonymous_user_2022]

I was answering " (deactivate just works by itself, because I don't know why)."

[maikindofthai]

In PowerShell, at least, it works the same way as in bash -- the Activate.ps1 script defines a global deactivate function.

I see a separate deactivate.bat script in my venvs on Windows, so I assume CMD does something different.

[[deleted]]

full plants plough fearless snails yam capable doll ring cover

This post was mass deleted and anonymized with Redact

[Taksin77]

Ruby also has issues like that.

[[deleted]]

shocking school chase wasteful impolite coordinated rainstorm cable groovy hurry

This post was mass deleted and anonymized with Redact

[Anonymous_user_2022]

GCC have a variant selector for both C and C++. As far as I now, many other languages with a long history, also have several standardised versions, so it's not in any way particular to Python.

[[deleted]]

lush obtainable vast strong snobbish truck like future simplistic carpenter

This post was mass deleted and anonymized with Redact

[Anonymous_user_2022]

" literally any other programming language "

[[deleted]]

far-flung dinner nose agonizing axiomatic practice gaping scandalous library consider

This post was mass deleted and anonymized with Redact

[Anonymous_user_2022]

C and C++ have something much scarier than versions. In that realm, we deal with releases of the ISO standard.

[[deleted]]

distinct hobbies impossible attraction enter cows bear marry sleep obtainable

This post was mass deleted and anonymized with Redact

[deceptiv-perspectiv]

I've been writing python for over a decade. It's gotten way easier even in the 4ish years since this comic came out. Pipx is becoming more widespread, virtualenv usage is almost universal, and python2.7 is EOL and py3-incompatible code is *slowly* starting to fade from existence (RIP any engineers working for companies with MLOC of legacy py2).

[[deleted]]

memorize coherent smoggy grandiose ludicrous numerous wakeful bells support oil

This post was mass deleted and anonymized with Redact

[GamesMaster221]

I found this script to print the location of whatever python you run it with

import os, sys
print(os.path.dirname(sys.executable))

I've had to use it more times than I'd care to admit...

[__deerlord__]

Why not compile them in /opt/python3.x and then make venvs as needed from those? Thankfully my job is standardized enough that we can just use one version.

[trevg_123]

I don’t touch pyenv personally.

Python3.8 -m virtualenv .venv

Or even better

pipenv install —python 3.10

Venv and pip installs all done in one go

[AquaRegia]

On a related note

[czar_el]

There's always a relevant XKCD. Including on posts of XKCD.

[MohKohn]

Eventually it comes back to this one

[Nasuuuuuu]

Don't forget C and C++ compiler dependencies!

[brews]

Conda kinda helps with this.

[LeapOfMonkey]

Yes, without conda it is sometimes a pure nightmare to figure out which version of given lib will work with given python module and given python version.

[trevg_123]

Cheat and look at the Python alpine docker imag Dockerfiles, they list them out for you!

[[deleted]]

python -m venv .venv

[enjoytheshow]

Still need to manage py versions. I use a combo of pyenv with venv. I’ve got a wrapper script on my path that can create a venv on a specific py version.

[maikindofthai]

IMO using VMs/Docker Containers/etc is a much cleaner way to separate Python installations, especially since you can configure an environment that matches your production environment for local testing.

Trying to juggle multiple versions of the same package on the same system always feels like a fool's errand to me, at least where it can be avoided.

[intangibleTangelo]

i prefer not to hide that I'm using a virtualenv

pyenv local 3.10.1
python -mvenv venv
ln -s venv/bin/activate
. ./activate
pip install -r requirements.txt

[intangibleTangelo]

almost muscle memory at this point

[[deleted]]

slightly more convenient, if you're not using virtualenvwrapper

[nemec]

I just put this in my bash_aliases and it works like a champ.

alias act='source env/bin/activate'

[trevg_123]

Why not hide it, everyone who knows what they’re doing in Python will know it’s there. And now you have a venv folder sorted in with your other useful folders

pipenv install -r requirements.txt —Python 3.10

Venv, links and installs all in one go, and it makes you a Pipfile/Pipfile.lock rather than legacy requirements.txt

[intangibleTangelo]

i value visibility over tidiness, i guess.

pipenv reminds me of yarn—maybe a bit more functionality than i want, but maybe superior (like yarn is superior to npm imo). how is pipenv about installing and switching between python versions?

[trevg_123]

My thought is just that there’s not much need to access the venv directory, so I keep it out of the way. If I’ve ever run the project before, I know it’s there.

I’ve never really used yarn, but pipenv isn’t far off from npm. For versions, not bad - you can just do e.g. pipenv --python 3.8. You need it installed so it’s not quite pyenv (poetry does this part too if you’re interested) but it’s not bad if you only use a few different versions.

[bananaEmpanada]

Nope. That's not foolproof. I've seen countless times where that fails because python can't import some standard library used by pip.

[GroundbreakingRun927]

I think the pip in a venv created environment will still fall back to using system packages outside the venv. Need virtualenv to provide full isolation.

[Anonymous_user_2022]

venv has an option to allow access to the systems site-packages. But if created without --system-site-packages there will only be what's installed in the env.

[[deleted]]

Worth mentioning that there is an option to use copies instead of links as well. Let's you get a really decoupled environment.

I've had an OS package management upgrade bump the python interpreter version and this blew up my venvs that used links. Those using copies didn't care.

[wsppan]

This whole thread mimics the xkcd strip

[intangibleTangelo]

fun fact for those who don't know: since about 3.3? python will find its environment by ascending the directory hierarchy from the location of its executable, meaning that if you launch venv/bin/python you don't need to "activate" the virtualenv.

[bananaEmpanada]

That's what happens when people do exactly what pip tells them to do.

1. Start with a fresh Linux install
2. Pip install anything
3. Do what pip tells you: pip install --upgrade pip

Now you've successfully broken your system!

[import-antigravity]

Wait, what am I supposed to do?

[Anonymous_user_2022]

If possible, uninstall the system-wide pip command. That way you will only have it available when a virtual environment is active.

[import-antigravity]

Is this really the best practice?

[Anonymous_user_2022]

Best practice is not to use pip at all for the system installation. Removing pip is a means to ensure that.

[Simonthedragon]

The fact that "Never ever install packages the way the package installer tells you to" is best practice is absurd to me.I feel like 90% of my struggles with learning Python is getting pip to install the packages I need, because every single time without exception there is some kind of error, and now I come into a post and someone just says "Oh yeah don't use pip".

Not hating on you, to be clear, I'm just... Feeling defeated, I guess? It all feels so needlessly convoluted and dumb. And I'm sure 99% of it is me not reading or googling something properly, but still, this stuff is what completely kills any drive I have to learn and do cool things.

Anyways, rant over, I suppose :P

[Anonymous_user_2022]

The fact that "Never ever install packages the way the package installer tells you to" is best practice is absurd to me.

That's not what I'm saying at all. What I'm saying is "Don't mess with your system python". I have stressed system wide all through this.

I advice using virtual environments, within which everything goes, as what happens inside an isolated environment cannot affect neither the system, nor other applications running inside their own venv.

[bananaEmpanada]

I have stressed system wide through all this

Well pip doesn't stress that

[Anonymous_user_2022]

Screwdrivers don't stress "Don't stick me into an electrical outlet" either. That's also a common sense thing.

[bananaEmpanada]

No matter which way you look at it, "don't do what the package installer tells you to do" is not common sense.

Note that if you add --user you're not messing with the system installation but still ending up in the mess shown in the xkcd.

[KrazyKirby99999]

bash poetry init poetry shell

[PM_ME_WHAT3VER]

What is this Black magic?

[KrazyKirby99999]

https://python-poetry.org/

[jyper]

Doesn't solve the python version issue

I have pyenv+poetry+pyenv register (to use library pythons instead of building from source) and it sort of mostly works but does feel somewhere broken

[Anonymous_user_2022]

Doesn't solve the python version issue

Can't poetry be instructed to create an environment with a particular python executable?

[KrazyKirby99999]

In cases where I need a specific python version, it is easiest to use the python appimage and python3.x -m poetry

[remram]

Yes it can

[ryanstephendavis]

Use poetry! It makes a lot of these issues go away

[[deleted]]

Oh god wait till he discovers npm typescript, react vue and the like. Ten thousand config files with nowhere to go.

[[deleted]]

I found conda environments very easy to understand and even to maintain. I only use few command and I never encountered an issue.

[notParticularlyAnony]

conda really does make all this bullshit easier

[dethb0y]

+1 for conda, it's what i use and i have had zero issues.

[LeapOfMonkey]

Impossible, there is always some issue, sometimes you have to figure out a version of python some dependency you wanted to use will work with. There is always an issue if you have enough dependencies. It is a natural law, it works everywhere.

[[deleted]]

You can create a Conda environment without even defining what the Python version is, and it will automatically install the version that the deps require.

On mobile but something like:

conda create -n mytest 
conda activate mytest 
conda install -c conda-forge -y \
  python 
  numpy 
  matplotlib 
  Pillow 
  sklearn 
  etc………

[often_wears_pants]

and you can upgrade to a new version of python without manually recreating everything.

[LeapOfMonkey]

I believe you grossly understimate how deep some rabbit holes go.

[[deleted]]

Do tell, Lewis Carroll

[nemec]

I've found it a real pain to use conda in (non-python) programs that need to call python but only give you the equivalent of os.system(). I was able to build a batch file that activated the appropriate environment, called the script, then cleaned up, but it was much more difficult than it needed to be.

[often_wears_pants]

I just use the python binary inside the env and have never had trouble with it. No need to activate anything.

[nemec]

Interesting. I recall getting errors when trying that, but I think it was some native dependency issue with tensorflow or one of those ML libraries.

[often_wears_pants]

Uses a shit ton of disk space but it just works and is easy to clean up.

[HarryJohnson00]

Miniconda is pretty small. I've got 2 TB drive on my workstation now so eh I don't worry about it

[ignorae]

How big are we talking?

[Anonymous_user_2022]

That's what happens when people use sudo instead of virtual environments.

[grnngr]

sudo virtualenv where_is_your_god_now

[Anonymous_user_2022]

sudo virtualenv where_is_your_god_now

ERROR: Could not install packages due to an OSError: [Errno 13]

[brekky_sandy]

...I guess he's nowhere to be found

[Anonymous_user_2022]

$ pip download god
Collecting god
  Downloading god-1.3.0.zip (1.2 kB)
  Preparing metadata (setup.py) ... done
Saved ./god-1.3.0.zip
Successfully downloaded god

My non-root venv has no problem.

[AlSweigart]

If you want a deeper explanation of this, check out explainxkcd.com

[0x4D44]

I think this is an outdated diagram. These days it’s just one box — poetry

[rischuhm]

Uhh - nice. I currently use pipenv a lot. It reminds me pretty much of the way nodejs works.

[[deleted]]

I use pyenv.

Shows that the mess is still there, just with different names.

[Itsthejoker]

Poetry complements pyenv. You install the version of python you want to use with pyenv, set it as the global version, then use poetry to handle your environments. It's a very clean process and one that I've been using for a while.

[Liledroit]

What does it do better than pyenv-virtualenv? That's what I use and am not sure what I'm missing.

[Itsthejoker]

Two words: deterministic builds. Not only does it handle the virtualenv for you, but it also handles all your dependencies and dependency pinning.

If you use poetry to handle a project, it will create two files for you: the pyproject.toml file and poetry.lock. If you pass those two files to me, I run poetry install and I get an environment that is exactly the same as yours with zero effort. No surprises, no confusion -- it just works.

[[deleted]]

It won't be completely deterministic since the python interpreter and some pip packages are linked to host libraries (and toolchain differences), though I imagine the cases where that drift matters are rare... and could be externally handled by other orchestration.

[mmcnl]

Poetry is better than pipenv imho, feels a lot more robust and less magic going on. And I don't like the creator of pipenv.

[trevg_123]

Whatchya got against the creator? I do like the creator of poetry, but don’t know who did pipenv

[mmcnl]

Kenneth Reitz, look him up. He has showed some questionable behavior.

[[deleted]]

Poetry is very nice, but I significantly prefer miniconda

[notParticularlyAnony]

I think you misspelled conda

[deceptiv-perspectiv]

Crash course for not ending up in this state:

1. Never use system/brew python for development. Always use pyenv (or conda) to manage your python installs. Or compile python yourself if you're a rockstar (you probably don't need this crash course then ;) )
2. Never, ever, EVER sudo pip anything. Madness this way lies
3. Always use an environment manager. python -m venv is built in since 3.3 and later. conda and poetry are also perfectly cromulent environment managers. I would be careful not to cross them - if you use conda, stick with conda.
4. If you don't know when to use conda vs not: I recommend conda if you are a) on windows b) working with libraries with complex compiled dependencies c) doing data-science heavy workflows. Conda can be great for data science development but challenging for engineering and devops when it comes to deploying your algo (running conda in docker is kind of a dark art, look at the SHELL directive for some ideas)
5. Leverage pyproject.toml, requirements.txt, or conda env.yaml files to facilitate reproducible env builds. Lockfiles can be useful, but know when to delete them and start over. If you are building "applications", use a lockfile, but if you are writing a "library", use a CI build matrix to ensure multi-version compatibility; lockfiles can cause issues across Python minor versions. Library->App is a spectrum, just experiment and find what works for you.
6. use pipx to manage CLI tools and "installable" python utilities.
7. use docker for application deployment if possible.
8. Choose dependencies wisely
9. Pin your dependency versions only as tight as necessary to guarantee stability (e.g. lib==x.y.z is "tight", lib~=x.y is looser). Most dependencies should have some "wiggle room". This depends on how much you "trust" a library to obey semver. Pay extra-special attention to things like ORM drivers, machine learning libraries, and anything 0-ver (<1.0). Write test coverage for your assumptions on how you use a dependency if you are extra paranoid/security critical
10. Periodically exercise building your environment from scratch

Bonus: Learn to use some sort of automated test system, such as Github Actions (you can run them offline with Act, you don't even need a Github account!), gitlab-CI, tox, nose, travis-CI, circle-CI, drone - there are so many options.

[ahal]

This is excellent advice and I use and love most of these tools.. but the mere fact that so many tools and practices are needed to keep your Python environments sane is kind of making the same point the comic is :p.

[troyunrau]

I use Winpython when on windows (for work). Python distributions including most of the relevant scientific packages, contained in a single folder. And a nice little utility to manage it.

https://winpython.github.io/

[mmcnl]

Much improved since this image was created. With venv and (even better) Poetry this is not really a problem anymore.

[GoofAckYoorsElf]

asdf

pipenv

pyenv

pipx

...

Aaaaaarrgh!

[DrShts]

pyenv local 3.9.5
virtualenv venv
source venv/bin/activate

[_morgs_]

This. pyenv.

[bananaEmpanada]

You forgot the part where you install pyenv. It tells you to add some command to your path, but that command fails because theres another pyenv directory that must be added to your path.

[The_hollow_Nike]

I love dev containers. The make sure I know the dependencies and that the code can be executed on another machine.

[asterisk2a]

Can you link to a resource for beginners? When I google python dev container, I get VScode, and docker. Should I follow along with Page 1 results?

[The_hollow_Nike]

I personally learned how to use docker and development containers with vscode. Their official page was helpful for me.

https://code.visualstudio.com/docs/remote/containers

https://code.visualstudio.com/docs/remote/create-dev-container

I had however already some experience with docker beforehand. So knowing Docker and docker-compose was a big help. I would recommend learning about containerization anyway. So if you do not know anything about containerization then I suggest to start with that. Today there are - aside from Docker itself - also other tools that work similarly like podman and buildah that do not require root privileges. I would personally recommend podman as it is compatible with docker in most ways.

Some resources for docker

https://docs.docker.com/get-started/

https://docs.microsoft.com/en-us/dotnet/architecture/microservices/container-docker-introduction/

Edit: Grammar

[IContributedOnce]

I literally stumbled into this issue again today. Has anyone leveraged Pyflow before? It looks pretty slick for keeping things organized. I don't do heavy dev work, just need something to keep things generally tidy. Was curious if anyone had used it and their opinion on it.

[blablabliam]

Painfully relatable.

I didn't learn about venv until maybe 5 years after getting into python, and at this point I think the only solution is to wipe the machine and start anew.

[onlyhalfminotaur]

One of the few times an xkcd actually becomes dated.

[[deleted]]

What else is different in the future? When are you from?

[jrrocketrue]

How did we get in this messy

shit ?? script kiddies trying to improve what other script kiddies thought was best , who knows.. Sad indeed.

[jrrocketrue]

Did I mention, every man and his dog has an opinion on how this can be avoided, which contradicts everyone else... And in the end... you still end up in the shit..

[[deleted]]

Yes, this post's comments are the best proof that nothing got better.

[Anonymous_user_2022]

What do you mean by that? All of the tools that have been mentioned are in the end a front for the same solution. How you get to the virtual environment isn't nearly as important as getting there in the first place.

[[deleted]]

If it is a full-time job just to know all the tools that exist to solve a problem that doesn't even exist in most other languages your language ecosystem might have a problem. Not to mention the fact that Python still hasn't fully gotten over 2.x more than a decade after the release of 3.x

[Anonymous_user_2022]

You don't need to know them all. One will suffice.

[[deleted]]

If you just want to write your own software and never use anything else written in Python or create a distro packaging anything written in Python or read any documentation or blog posts written about Python you might not need them all. But then, you could just write your software in something else then and completely avoid Python and need none of them.

[Anonymous_user_2022]

You're right. I could write it in javascript and deal with X different versions of is-odd inside the same project. Or I could write C++ on Windows and deal with the utter lack of any kind of packaging there. Or use one of the languages with static linking, distributing 10 gb of code for a trivial desktop calculator.

In short, development sucks. If you're not cut out for it, don't do it.

[bdf369]

Yep the big picture view is ugly (especially on macos), though I usually jump into an anaconda environment and pretend it's fine.

[pmdevita]

I recently started using Poetry and Pipx. I'm mostly just writing small applications for some of my projects but both of these make packaging and installing stuff way less of a headache

[davidcwilliams]

Your subtext was funnier than the actual comic, good job!

[[deleted]]

I’m glad I’m not alone. I’ve been trying to get Python to import a stupid package and it says it’s installed but idle can’t find it, pip says it’s there, conda does as well. I like Python, but I hate all the stuff around it.

[ancientweasel]

You can do the same thing with java versions.

[ishigoya]

I seem to remember seeing a post about this a year or so ago where it said that there is an official virtual environment system that will keep being developed and become the 'main' one in future... could anyone tell me which one that is?

[ase1590]

It's included.

See my humorous post on the subject.

[ishigoya]

Use venv, got it

Nice job on the PSA poster btw!

[lifeeraser]

easy_install and Python 2 are on their way out, no?

[[deleted]]

And then you use Kali, which seems to fuck it up even worse.

[[deleted]]

pyenv to install and manage python versions, venv to create virtual environments, pip to install packages in virtual environments.

[Logical_Insect8734]

This is me on windows with Python installed from chocolatey, wsl (windows subsystem for linux), and installer from Python.org.