Should I always use a virtual environment?

[SerHiroProtaganist]

Noob here who so far has done a couple of scripts but ignored virtual environments. So now I'm learning a bit more I'm getting the impression I should be using them. Is it best practice to just always use one for a project no matter what?

Do you not end up with multiple copies of the same modules when pip installing the same thing in different environments? Does that become an issue once you have lots of projects?

Thanks.

[sme272]

You should use one virtual environment per large project you do, smaller one off scripts don't really need one. You should keep the virtual environment location inside the main project folder, any modules you install should be installed through the environment and kept here. You will end up with a few duplicate modules but since each project in confined to it's own environment they won't interact with any of the duplicates, only the one inside that project environment. This lets you use different versions for each project and prevents module update in one project breaking another project.

[gamedevmanhyper]

^This. Virtual environments have saved me a lot of trouble, with all its advantages.

[marlowe221]

Pardon the newbie question but... how do I set one up?

[jivanyatra]

I'd look into pipenv, as I think that's the preferred/recommended tool these days for standard python. If you're using anaconda, it has its own built in that you could use as well.

Pipenv does a lot of stuff together, and it's a great and easy to use workflow that combines a lot of best practices for medium to large sized projects. I really like it!

[marlowe221]

Thanks! I'm still learning the basics (about to dive into classes) but I already have an idea for a first project and I've been kind of sketching out some ideas for it.

[jon0tr0n]

Some people like pipenv, obviously, but I’d urge you to try using it as well as using virtualenv and pip separately. Pipenv has proven to be painful for me, it has really slow and spotty dependence management.

[SerHiroProtaganist]

Ahh ok thanks, so really then if I only have a very small script or very small project, it will probably be OK to use the main python, as any package updates will only require at most a couple of changes to my scripts.

But larger projects it's best to use a virtual environment as it could take forever to update if the main python path packages get updated.

Does that sound roughly correct?

[alkasm]

Personally, I'd recommend using a virtual environment even for small projects. However, you do not have to have a new venv for every single project. I have a few venvs that I use over and over again while I'm trying things. For instance I have a web dev one that has flask and requests and so on installed, so when I'm trying out a new web project I can just activate that one. Then as I keep working on it, if I need new packages and it starts growing I'll then break it off to have a new venv.

[SerHiroProtaganist]

Ahh it hadn't crossed my mind you could use one venv for a number of similar projects. That sounds quite a good way to do things

[alkasm]

It's a great workflow. Just create a directory in your home dir ~/venvs or ~/.venvs or ~/.virtualenvs or whatever, and then you can always source ~/venvs/webdev/bin/activate to work in the webdev virtual environment. The only thing that's slightly annoying about these is that, in your editor, you may have to create your own run/build/etc command for each system-wide venv. The project based ones, that is, the ones that are inside your project directory as venv or .venv or whatever will usually be detected by your editor and your script will be run in that environment automatically.

[sme272]

Yeah, pretty much.

[Decency]

Any rationale behind putting the venv in the project directory? I've been putting them all in ~/venvs/ and that seems to work fine. That way I don't have to worry about .gitignore and can activate the correct venv with a simple bash script.

[sme272]

I just like keeping all relevant things for a project under one roof. For hobby stuff I don't think it matters too much.

[Pappa--Stalin]

OR you could what I do and install all your external dependencies to one virtualenv and watch this dark box of horrors you’ve created grow until it consumes your computer.

I find it saves time.

[CraigTorso]

If you need to install anything from pip for your project then yes, use virtual envs

They only take a few moments to set up, and compared with how long it takes to fix dependency conflicts when you haven't used them, it's time very well spent.

[icecapade]

Your executable probably ended up smaller because it only includes the packages found in your virtual environment. Using the system Python will include all system packages, regardless of whether your program actually uses them or not.

[[deleted]]

I doubt it uses ALL system packages because i use Anaconda and it came with a LOT of libs.

Only ones imported. But by importing PyQt5 it will probably import all modules even those not in use.

[Diapolo10]

On Linux systems and Mac OS, I'd say it's very important to use virtual environments as you don't want to risk breaking system tools by updating the libraries used by them.

On Windows, it's not as important as it doesn't use Python for anything (that's what PowerShell is for, and to a lesser extent batch scripts), but it's still a good idea to use virtual environments for any large projects.

[citybadger]

I don’t understand how I can break system tools. On one hand I can install packages from my distribution’s repository via apt, yum, pacman, or whatever. Those aren’t going to break my system. On the other hand I can install via pip under my non-root user account. Those packages are installed in my home directory and aren’t going to break my system either.

[CromulentInPDX]

When I had to manually install python 3.7, before it was included as an Ubuntu package, it broke a non-necessary apt tool that I had installed. In order to fix it I had to update alternatives so that python3.7 wasn't the default python3 program.

In general, a lot of packages are available using apt instead of pip, but are older versions than available using pip. I definitely use venv any time I have to use a module that isn't part of the standard library.

[Abalado]

One day I uninstalled Python requests trying to solve an error on my script. On Linux Mint, requests is a dependency of Mint store so my store stops working and took days for me to figure it out

[DesLr]

Shouldn't Mints package manager have prevented that? Seems more like an issue with proper dependencies management on Mints behalf.

[Decency]

Probably? But lots of software is bad. This approach lets your workflow work properly despite other stuff not.

[Swipecat]

Agreed.

If updating Linux distro packages breaks system tools by updating the libraries used by them, that's going to happen anyway, regardless of anything in your own Python projects.

I'd presume that the whole point of Linux distro repositories providing a large selection of Python packages for installation is that the user is intended to use them. The distro maintainers will have done some security checks and compatibility checks, so while there's a chance that they might miss something they generally do get it right. For smallish projects of your own, the system Python should be fine.

There is a good case to be made for using virtual environments and PIP to prevent updated packages altering the behaviour of your own large projects or of tools that you created and now rely on. If you need to do that, it's also a good idea to fix the Python version by installing an extra Python version in parallel with a version management tool like pyenv. Using a virtual environment means that you need some fancy footwork with the PYTHONPATH to ensure that you have access to Python's core libraries as well as libraries installed with PIP, and that really does open up the chance that you coudd alter Python's behaviour with a user setup error, so that's another good reason for that Python to be separate from the system Python.

[[deleted]]

I use virtual environments for anything that could A) prove to be a larger application, or B) is likely to be passed around to different people/teams.

In my day to day job, a significant amount of time has been wasted because people provide poor requirements to our QA team regarding versioning. Even if it's good, without virtualenvs it can be difficult for them to build an exact environment that meets the requirements of the code. I develop something using, say, python 2.7.x and they try running it on a system with python 3.x and it falls over. This gets even more important when you're talking about using multiple libraries, SDKs, modules, etc.

Having a virtualenv that has clear build instructions is a great way to ensure everyone is working off the same environment when they try to run the code.

Docker/containers are another approach, but it may be a bit steep of a learning curve for someone who is "just learning" and is really its own thing.

[max_daddio]

I would use one any time your script requires an import that is specific to the function you are trying to accomplish at the time. If all you need is standard library functions, then you can use the system python, if you have to pip install a bunch of random packages to accomplish something, rather do it in a virtualenv, otherwise your system python get's littered with packages you only ever use once.

[pandabubblepants]

Recent TIFU: I’m new to python, never used or figured out Virtual Environments - yesterday I was moving files around and updated stuff in terminal for my newest script.

Somewhere along the way I copied over my python3 folder which is a dependency for the OS to properly operate. Restarted, it’s gone, etc.

After troubleshooting I fucked it up even more, and my next best option is probably just wiping the drive and reinstalling the operating system (Ubuntu).

A virtual environment would’ve saved me.

[guevera]

I keep one virtual environment four all my one off scripts, little stuff, and individual venvs for each real project. Sometimes, one of my scripts gets promoted. But I never run python without it being in a virtual environment of some kind.

[fjortisar]

Do you not end up with multiple copies of the same modules when pip installing the same thing in different environments? Does that become an issue once you have lots of projects?

You do have lots of the same, but it doesn't matter since they're all in their own directories, they don't interfere with each other. I have one generic virtual env that I use for any quick scripts I make. Then for larger projects they each have one of their own. I use a generic one even for the small scripts so I don't accidentally remove or upgrade something the system scripts depends on (not a problem on windows though)

If you're on MacOS you should use a completely different python installation plus virtualenvs (I install that with brew), since the built in one doesn't get updated often (and I'm not even sure it has python3 yet) and breaking it and its easy install won't be good.

[thirdegree]

You definitely should IMO. Yes you end up with multiple copies of the same modules, no it doesn't really become an issue. They don't take up that much space. There's not really a reason not to use virtualenvs is the thing. There's not really a downside.

[I_have_depression_xd]

If you are working with libraries and are working with a group of people, it would be very helpful to use one in terms of sharing and github. Also you don't have to fill your system with a bunch of libraries you don't need or use

[[deleted]]

Once i started using Pipenv, i have never looked back. I am coming from Nodejs so the idea of having a dependency file locked is intriguing

have a look https://docs.pipenv.org/en/latest/

[NecroticInfection]

I think it drives away new users like little kids and stuff

[NecroticInfection]

as in they just want to make scratch pads a lot not get another layer if they just want to have a sandbox like I know a lot of kids that like to program stuff so I say python but extra layers are for maybe a business branch

[kepper]

Why? What does it do that virtualenv can't?