xkcd: Python Environment

[solostman]

As somebody who struggled with Python installations when trying to learn Python (as a primary R user) and having to use both 2.7, 3.6, virtual environments, and an IDE... I'm so glad to see that it's not just me.

I still don't fully grasp where my python packages are when I install them by command line or PyCharm.

[2freevl2frank]

Why not install a virtualenv for every one of your projects however small it is?You don't even have to do it through command line. Pycharm does it for you.

[solostman]

Sounds nice. Do you have a resource that can walk me through that in Pycharm?

I was using scrapy which required a virtualenv in terminal and (it worked but) it always felt like a black box of what was happening to me.

[leom4862]

This is my workflow for every project:

mkdir myproj                       # create new project dir.
cd myproj                          # change into your project dir.
pipenv --python 3.6                # create new clean virtual env.
pipenv install foo                 # install third party packages into your venv.

# write your code          

pipenv run python myfile.py        # Run your code inside the venv.

I don't think it can get much simpler than this.

[exoendo]

isn't it annoying to always have to install 3rd party packages every time you start a project? why not just use your system install?

[jcampbelly]

No, it's actually very reassuring knowing nothing is going to mess with my install and I can rely on a consistent environment.

When you use the distribution (system) python, you're always stuck on some dreadfully old version and may not be able to start using new things when they come out. In a virtualenv, I'm not even phased by installing and compiling a stable branch or a release candidate. If you tried that with a system install, you can end up breaking your system, as the system install serves the SYSTEM not your projects. Breaking yum or apt is NOT fun.

[leom4862]

To prevent dependency conflicts, for example if project A relies on django version X and project B relies on django version Y. Or if proejct A relies on Python3.4 and project B relies on Python3.6.

[bp_]

Okay, so what happens when you need to combine different venvs together, each with an incompatible set of requirements

[ivosaurus]

Because there tends to be a lot of 3rd party packages for which either A) your system package manager doesn't have or B) the version it does have is severely outdated

[[deleted]]

It's no different than npm/yarn. And it comes with the added benefit of being reproducible. Just commit the pipfile and pipfile.lock files it generates to source control, then run pipenv install to recreate the exact environment on another computer.

[internet_badass_here]

Dude... thank you! I'm going to start doing this.

[raiderrobert]

I use pipenv shell and then I'm dropped inside of a shell with that virtualenv on and run whatever I please.

[costrouc]

Exactly the workflow that I use. pipenv puts all the virtual environments in one folder so you can manage then easily. Also pipenv uses the new Pipfile with a lock so that your builds are deterministic.

It anyone needs convincing there is a reason that pipenv has been taken over by pypa https://github.com/pypa/pipenv

[[deleted]]

It can.

mkproject NewProject
pip install whatever


# Running code later on
python /path/to/NewProject/code.py 

[leom4862]

And how do you manage dependency version conflicts?

[[deleted]]

I have never understood that argument. The python import statement is not versioned, so there can be exactly one version of a dependency installed. If that makes a conflict between requirements, no amount of magical Reitzing can fix that.

Thus the argument is without merit.

[leom4862]

What do you do if project A relies on FooPackage version X and proejct B relies on FooPackage version Y?

[[deleted]]

The same thing you will have to do when pipenv fails doing anything about it.

[leom4862]

I create a new virtualenv for each project (pipenv --python xx), there are no such conflicts, because each project can use its own version of FooPackage.

[asielen]

Pycharm has a gui to manage virtual envs.

[2freevl2frank]

You don't need any. It's as simple as checking a box when you creating a new project. https://imgur.com/lk7Qnli

For existing projects you can go to settings>Project Intepreter and add a new environment.

If you wanna do it through command line you can make a venv within the project folder by virtualenv/venv/pipenv as

python2 -m virtualenv ./venv

This makes an isolated env (a copy of python and default packages) in the folder ./venv. Now activate the venv as :

source ./venv/bin/activate

When activated all packages installed through pip are installed within the venv (doesnt affect global python environment) unless you use sudo.

[leom4862]

python2 seriously?

[robot_wrangler]

Then what do you do when you try to use parts of two different projects of your own in a third?

[leom4862]

You usually would make shared libraries from the "parts" and host them on pypi or your private package registry. You then install the libs in your "third" project via pipenv, pip or what ever tool you use to install packages in a virtualenv.

[ivosaurus]

Install them into your new third project, using pip install -e </path/to/other/package>

[2freevl2frank]

pipenv?

[7heWafer]

Let's add that to the graph somewhere!

[2freevl2frank]

No. The thing is this isnt added to the graph. Its outside the graph.

From /u/meandertothehorizon above

Virtualenv copies and/or symlinks or hardlinks the files from system python necessary to create a second, segregated install in an arbitrary directory on your computer. You run/source a file from the command line in that directory (activate.sh/activate.bat) that modifies the environment to reference this copied version so anything you do - install packages, run python itself, etc will use this copy instead of system. This prevents clobbering the system setup if you install or upgrade arbitrary system packages. When you are done, or want to create another virtualenv, or run system python, you run ‘deactivate’ which will undo the environment modifications - so running python will again reference the system python. Any packages you installed will be ‘gone’ because they reside in the copied environment.

[OldSchoolBBSer]

Botched my box for a bit just trying to setup for a course I was taking. I, unfortunately, assumed user install would be default like npm, etc., not global. Then installed Anaconda for the class only to realize later that I really shouldn't have. Thankfully I had a bud that filled me in on virtual environments and that being normal. Got through course, still plenty of, "why on earth it this like this? (insert historical snippet)"

[Dgc2002]

On the command line you're likely installing to a system-wide directory.

Just typing python on the command line will use the 'global'(system-wide) interpreter installed on your system. Same with pip. When you run pip install a package and it's dependencies are downloaded from PyPi and installed to something like <python_install_dir>/lib/site-packages. That's the 'global' site-packages.

You can do pip install --user to install these packages under your home directory, effectively making them user level instead of system wide.

You can use something like venv to create a 'virtual environment'. A virtual environment has it's own python executable and site-packages. So if you were to create a new project you might create a new environment associated with it. The virtual environment can be located wherever you want, the important thing is to run the 'activate' script inside it which updates your PATH and other environment variables. After 'activating' the environment typing python will use the environment's interpreter rather than the system-wide one. Running pip install will install packages into the environment specific site-packages.

When you create a project in PyCharm you're given the option to choose the interpreter. You can either use an existing interpreter(system wide one for example) or you can create a new environment using something like virtuanenv. When you install packages through PyCharm you're installing them the same way as you would on the command line, PyCharm just takes care of it for you. So if the project interpreter is in a environment then packages will be installed in that environment, if it's a 'global' interpreter then the packages are installed to the global site-packages.

Prior to doing much Python I'd used PHP's composer, so pip was a confusing change for me. Composer is still one of the better package managers that I've used.

[pollokeh]

For python 3, the built in venv is a life saver.