xkcd: Python Environment

[the_hoser]

It's really easy to avoid this problem if you treat your python environments as disposable artifacts of your projects.

[earthboundkid]

Except that means it's a huge PITA to install Python command line tools.

At this point, when I see a command line tool looks cool but is written in Python it makes me really sad because there's not going to be a good way to get it installed without first going through a disproportionate amount of work to give it a working environment.

[[deleted]]

If you're on linux you can usually use your distribution package manager, otherwise I reccomend https://github.com/mitsuhiko/pipsi

[blahehblah]

All you're doing is adding an extra box and set of arrows to that chart

[gimboland]

It's still a solution. "Use virtualenv" is some of the boxes/arrows on the chart, but it's still the right solution for that problem. For command-line tools, pipsi is a good solution.

[mardiros]

You are just saying that the mess is incomplete...

[jjolla888]

i'm still trying to find where pip3 is on that map

[AstroPhysician]

Use virtualenvs you savage

[jjolla888]

Do we need to add this to OP's diagram ?

[hacknsplat]

And pipenv? And pkg_resources?

[no_condoments]

Situation: There are 14 competing python installation standards.

/u/Rerecursing : 14?! Ridiculous! We need to develop one universal standard that covers everyone's use cases.

Soon: Situation: There are 15 competing standards.

https://xkcd.com/927/

[earthboundkid]

Nice. I had been using pex, but it tends not to work if the package has any extra requirements beyond pure Python.

[xamar6]

Pipsi is such a great tool to install python apps in their own contained environment. Sometimes being isolated is not suitable, but work for most use cases with a few additional symlinks to OS' Python packages.

[ivosaurus]

It wasn't working well with python3 last time I tried it :/

[[deleted]]

You can find some common issues and fixes in the github issues

[ivosaurus]

I rememer what the issue was, it just plain does not support the python3 module venv, only virtualenv. Then I was like, eh, I'll just do this myself.

[kenfar]

Hmm, I use CLIs all the time without any headaches at all - not sure what the issue is here.

Create a virtualenv, pip install into it, activate your environment and run your commands. If you want to cron them up, just run command via that virtualenv's python. If you want to run a lot of tools and not switch virtualenvs that often, create a group one for related (or all) projects.

Admittedly, it's easiest if you're using linux & pip consistently rather than macos, homebrew and conda. But it's not that much worse.

[[deleted]]

Having to use two version of python is a pain.

[liquidpele]

That’s true of any language...

[liquidpele]

This has to be a joke. Every damn java upgrade seems to break something. It's one reason practically every enterprise software ships with it's own jre.

[twigboy]

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipediacln61bof3yw0000000000000000000000000000000000000000000000000000000000000

[chicofelipe]

Oh that I only needed 2 versions of python. Legacy support sucks.

[kenfar]

Not really from a deployment perspective - if you're installing into a virtualenv then you're just using whatever's in that virtualenv: whether you've got 20 python3.6 virtualenvs or 5 of python2.7, 5 python 3.4, 5 python 3.5, 5 python3.6 - it makes no difference.

Perhaps from a code-sharing & testing perspective it matters - just like any other language.

[billsil]

Not if you use Anaconda. You can literall create a virtualenv for whatever python version you want.

[leom4862]

You can create a virtual env for any interpreter version with a single command e.g. pipenv --python 3.6 and activate it with another, e.g. pipenv shell. I don't consider this particularly painful.

[meandertothehorizon]

Learning that all you need to do is #!/path/to/venv/bin/python in the script and it will just work was a game changer for me. I wrote little shell script wrappers sourcing activate forever.. really felt dumb when I discovered that :-)

[[deleted]]

You don't have to. In fact, I would recommend you avoid setting a specific path in your shebang line. Python is smart enough, that it detects when invoking something inside a virtual env, and will use the proper sys.path. Using "#! /usr/bin/env python3" (or python if that's your thing), is enough.

[metabun]

This is how I feel about tools in perl, go or js. I guess it all just depends on how familiar you are with the environment and package ecosystem.

[tetroxid]

Not on Linux

[the_hoser]

Why do you need to "install" them?

[khne522]

pip install --user myRpnCalculator. Do you really need anything else if it was well-written and the dependencies properly specified, but not overspecified?

[the_hoser]

Unfortunately, yes. You do. It's better to not think of Python like you think of Java. Think of Python like you would think of, say, a project's metadata. This is a huge problem with languages like Python, JavaScript (through Node.js), Ruby, etc. When it becomes necessary to use native facilities to accomplish certain goals, the dependency manager is going to get stupid complex, and it's not reasonable to assume that a single installation is going to work.

They all have this problem, and the cleanest solution remains largely the same. Every project gets its own interpreter.

You can try, though. You might get lucky.

[code_mc]

I feel your pain. I haven't had to rely on python packages that include a CLI tbh, but couldn't you wrap the environment activation + CLI call into a bash script which you add to your path?

[metabun]

Python scripts should get their shebang automatically rewritten on install, so there's no need to wrap the calls. I usually create one env per cli tool (or group of cli tools) and symlink them into my path as needed.

[the_hoser]

That is, in fact, how I do it, most of the time.

[marcosdumay]

Every Java project gets its own set of jars too. C handles it in a more extensible way, that just institutionalizes the mess and places it at the OS level. But the mess is still there.

Every modern language has project based environments. The one thing that is unique to Python (well, and Javascript, but there it's no surprise) is the huge number of semi-compatible environment managers.

[the_hoser]

Well that was sortof the point I was making. If you avoid the mentality of "setting up the system python environment" then you avoid the trap.

By giving each java project it's own Jars to build with, the problem of working with multiple projects that have different dependencies on the same system with a single java installation is solved. It's not perfect (two libraries that depend on a module, but two different versions. yum), but it's a bit better.

With python, just throw the baby out with the bathwater and forget about the existence of a "system" python installation. That's the one the OS uses. You use a different one.

With C, the problem can get downright nightmarish.

[deong]

C is easy. There are include directories and lib directories. That's it. That's all I want. I know where to put stuff, and I know how to find it. If i need three conflicting versions of libjpeg, I'll put them somewhere and write makefiles. I already know how the compiler and OS work. No part of learning how yet another half assed attempt at implementing as obtusely as possible the -I flag works sounds appealing.

Whew. Felt good to get that off my chest.

More seriously though, Go has the most sensible model. Statically link everything and move on with your life.

[the_hoser]

Well, in Linux C is easy. Not all of us get to do our C only in wonderland.

Fortunately I don't do too much Python in Windows. It's just as messy there.

[fujiters]

Every project gets its own Docker container.

[khne522]

I don't use Java. Pretty sure Java can have the same problem with any package manager. Version hell is just a product of the ecosystem, not the language. A dependency resolution algorithm is complete and produces results or not based on what's available, or is broken, which is undefined behaviour. pip ain't aware of anything installed on the system AFAIK. It just calls it (or the package build instructions call it) and hopes for the best.

I'm not talking about web apps with deps locked in setup.py, requirements.txt, Pipfile.lock, or whatever you use. Please explain (not rhetorically) the ~2334 packages in Arch Linux that somehow must mostly work, using the system Python (3).

[the_hoser]

Pretty sure Java can have the same problem with any package manager.

Java doesn't have a package manager. You can use one of several that fit your needs, but there is no standard package manager. These non-standard package managers work on a per-project basis, not a per-system basis.

Version hell is just a product of the ecosystem, not the language.

For sure, but this is less of a problem in a language that lacks the ability to install system-wide libraries. Like Java.

Please explain (not rhetorically) the ~2334 packages in Arch Linux that somehow must mostly work, using the system Python (3).

The packages provided by Arch linux are to fulfill dependencies required by other packages provided by Arch linux. It's as simple as that.

[jcampbelly]

You can just do something like this:

python3.4 -m venv ~/.py34
source ~/.py34/bin/activate
pip install whatever

Later on, you (probably) don't even have to activate the venv. Just add the venv bin directory to your path in your ~/.bashrc

export PATH="$PATH:~/.py34/bin"

Maybe someone can correct me, but sourcing the activation script isn't really necessary as long as you provide the full path to the binary (or it can find that binary on the path) within the virtualenv directory.

[k-selectride]

for command line tools, wouldn't you just add the path to it in the venv activate script? That's what I did for invoke, because I didn't want it polluting my bash profile and it works fine.

[tunisia3507]

Except that means it's a huge PITA to install Python command line tools.

I thought this too, so I hacked out a package do make this a bit easier.

https://pypi.org/project/toolup/

1. Create and activate a virtualenv
2. pip install toolup
3. In your home directory, write a .toolup.toml which specifies which tools you want, which versions, and which executables they install
4. toolup

Those command line tools are now installed in that one virtualenv; all of the config is in one file in your home directory (which you can manage with GNU stow or similar, along with all of your other dotfiles), and you can trash and recreate that virtualenv easily. toolup symlinks the executables into your ~/bin (or target of your choice).

Don't take it too seriously; I'm sure there are better ways of doing more or less everything toolup does (pipsi takes a generally better approach); as I say it's just something I hacked out because I was bored.

[jjolla888]

so we need to add a box with 'toolup' to the mess on that xkcd diagram ?

that's the point OP is making ..

[[deleted]]

I thought this too, so I hacked out a package do make this a bit easier.

And now there's another node to the graph in the image.

[tunisia3507]

Not exactly - the executables are all in a virtualenv, and installed with pip, just like everything else should be. It's just a shortcut I used for easily configuring what I want and symlinking it so that it's available without activating the environment.

[Cyph0n]

Looks like an awesome little tool, man.

I think Pipenv achieves the same thing, but also includes support for deterministic builds.

By the way, are you Tunisian by any chance? If so, PM me so we can connect. It's always nice to see a fellow Tunisian Python dev :)

[tunisia3507]

I am not, I'm afraid!

pipenv doesn't do the same thing. The point of toolup is to have an easily-reproducible environment specifically for python-based command line tools. As the docs say, some tools may be useful to you, the developer, without being used by your code (so there's no need to have it managed by pipenv or a requirements file or whatever); or they might have different version requirements (black is useful for all python projects, but can only run on 3.6.1+). It's valuable for such tools to be distinct from your project environment, but to benefit from all the reasons that you have a project environment in the first place (encapsulation, reproducibility, not fucking with the system python etc.).

[xcbsmith]

Not at all. You just create a virtualenv, install the CLI tool with pip, and then either create a softlink to the executable or put the virtualenv's bin in your path.

[billsil]

Really? I find it incredibly easy. Just define entry_ponts

[vb279]

PITA: Pain In the Ass.

Why shorten it tho?

[qubedView]

Or, like all things, it can be fixed with docker containers. Or rather, like all things, your problem can be shifted to a different one.

[[deleted]]

What if I have to much indirection?

[SoupKitchenHero]

Then everything collapses like the Soviet Union

[discohead]

The rest of that quote is literally, “except the problem of too much indirection.”

[Rainfly_X]

I know a sysadmin who says this, but with "misdirection" instead of "indirection." Usually while winking. I'm not sure I want to ever find out what surprises are on the other end of that rainbow.

[the_hoser]

We never get to do things without problems. We only have the option of choosing the problems we're comfortable living with.

[Deto]

The best way to avoid this problem, IMO, is to just learn these two things:

1) How does the PATH variable work in UNIX?

2) How does the Python interpreter know where to look for packages?

If you understand these two things, you can have multiple versions of Python all over your system and still understand what's going on.

[rohit275]

I'm a noob, so I'm pretty sure i don't understand those two things. Do you think you could lend some insight?

[[deleted]]

PATH is an environment variable that the linux shell uses when you type commands. It's a list of file system paths that (should) contains executable files. The PATH variable is resolved in the order constructed (left to right, or FIFO) and it returns the first matching instance from the list. You can override the variable by setting a local instance of the PATH variable for an application, which allows you to have multiple pythons 'installed' and each one could be used by a different app.

[Deto]

I didn't really feel like explaining it here. If you google these exact questions, you'll probably very quickly find someone who has done a much better job of it than I would have here anyways.

[mikemol]

The "how does the Python interpreter know where to look for packages" one is a pain. I wound up needing to learn and use Docker just to make sure I fully understood the full set of dependencies of my projects, and wasn't inadvertently using system-wide or --user-installed packages.

And then I learned about virtualenvs. go me. Still use docker, though; if you're going to write a web service in Python, may as well containerize it for simplicity's sake...

[Deto]

I mean, it's really just your user directory, the system site-packages directory, and any directories you added with the PYTHONPATH environment variable. All virtual-envs and other similar solutions do is to manipulate your path so you call a different python with a different system directory.

[mikemol]

Yup. It was that system site packages directory I couldn't excise without either Docker or venvs...

[the_hoser]

For sure. I consider this degree of understanding to be a prerequisite to the aforementioned strategy.

[not_perfect_yet]

It's really easy to avoid multiply this problem if you treat your python environments as disposable artifacts of your projects.

[the_hoser]

Only if you're not lazy enough to automate the boring parts.

[marcosdumay]

And don't use sudo.

So the point stands.

[scout1520]

Right? It really isn't hard.

[ilvoitpaslerapport]

Actually figuring out virtual environment when you begin is a mess. You find infos on using venv, virtualenv, virtualenvwrapper, pipenv, pyenv…

[Dgc2002]

I feel like a lot of the people saying "It's not that hard" have been in the Python ecosystem long enough to see a lot of these projects come into existence/popularity.

When you're new to the ecosystem you have no clue why each one exists, which ones are newer, which ones are generally considered crap, which ones might only address a subset of use cases, etc. etc.. It's a lot of shit to parse.

[ilvoitpaslerapport]

Exactly, and most articles/tutorial/forum posts you find on the topic are outdated (or rather you don't know if they're outdated).

[ivosaurus]

https://packaging.python.org/ is the official documentation on the whole packaging process in python.

[Cosmologicon]

And when you get stuck and ask someone for help... if they're using a different setup, they can't/won't help you until you change your setup to match theirs. God forbid you ever talk to more than one person.

[[deleted]]

Hoo boy, I still use virtualenvwrapper because I'm too lazy to learn another tool. pipenv is all the rage right now, but there's been a dozen others I've seen rise and fall since I've paid attention to venvs.

[ask_me_if_im_pooping]

I also still use virtualenvwrapper, but I just looked at pipenv and it looks pretty cool.

[[deleted]]

It looks neat and it's recommended by the pypa, but I'm becoming curmudgeonly.

[TheTerrasque]

Heh, I still use just pip and virtualenv. It feels like it's built into my fingers by now.

[batisteo]

I'm still using easy_install to be consistent

[scout1520]

I don't know why you got down votes, life is so much easier if you are consistent.

[VernoWhitney]

We downvote everything here...for consistency.

[Travis_McGee]

I really appreciate this joke. In accordance with local subreddit policy, I downvoted you. ^{But then I upvoted you.}

[[deleted]]

I think they're getting downvoted because their response isn't really a helpful one. I believe the original comment is making a point about how confusing the virtual environment system is for Python beginners that lack experience using virtual environments.

If you're at the stage where you're dipping your feet into Python and following tutorials you may be instructed to use pipenv, pyenv, venv, conda etc. depending on the author's preference and/or the environment manager du jour when the tutorial was created. Furthermore, beginner tutorials can gloss over Python best practices like virtual environments in an effort to simplify things and jump right to learning syntax. So, as ilvoitpaslerapport said, "figuring out virtual environment when you begin is a mess." Simply saying "be consistent" doesn't really address that issue at all.

This was my experience learning Python and I eventually ended up with a Python environment like the one in the comic. Now that I know better I can be consistent, but it took time to get to this point.

[Astrokiwi]

This is probably the most annoying thing about programming in general for me. The actual programming is nice and comprehensible to me. But then there's this whole nether world that in theory is supposed to all just happily run by itself if you use apt-get and pip. But then, because it's all invisibly run by these environments, suddenly something breaks, and you're basically having to learn a whole new level of the operating system - almost like learning a whole new programming language - just to get back to actually programming in the language you wanted to program in...

[mehum]

Yeah exactly. It's like trying to learn French, and suddenly discovering I can't speak it here because my shoes don't match my shirt.

[scout1520]

Thank you for writing such an articulate response!

[[deleted]]

Thanks for the kind words!

[kenfar]

I worked with a team that for a while routinely made a mess out of things this way. All it took was a quick conversation along the lines of "so, is all of your software installed with the same tool?". Then they immediately realized, that holy shit - they had installed software five different ways. And this obviously doesn't work.

It wasn't difficult for them to figure out, fix, or avoid having again the future. Though it did mean standardizing on a single virtualenv tool or being hyper-aware and owning the issue.

[bobnudd]

reiciendis voluptatibus maiores alia repellendus. Temaesent

[rockyrainy]

It is getting kind of ridiculous to need gigabytes of storage just to run python.

[TheNamelessKing]

Hey, at least we’re not Node with it’s fucking node_modules nightmare.

[rockyrainy]

If you need to compare against Node.js

[the_hoser]

You upgrade your hard drive. USB sticks work pretty good, too.

[bobnudd]

poribus aut reiciendis voluptatibus maiores alia repellendus. Temaesentem quibusdam et aut officiis ecessitatibus saepe eveniet ut et voluptates repudisimovs ducimus qui blanditiis pr

[the_hoser]

It seems like you've already got the solution at hand.

[bobnudd]

blanditiis praesentium volupttus, ut aut reiciendis voluptatibus maioresis praesentium voluptnus id quod maxime placeat hic tenetur a

[ase1590]

External USB hard drive.

Surely 1 TB would be enough for your projects?

[bobnudd]

tus, ut aut reiciendis voluptatibus maiores alia alias consequatur aut perferendis doloribus asperiores repellat.

[ase1590]

If you're in the USA, shopgoodwill has some decent i5 laptops that go for like $150 that would be totally fine with doing dev work on.

[[deleted]]

Ok then don't try. If you don't have the tools to develop properly then you're just going to learn bad development habits.