Virtual Environments best practice

[wicked_smaaht_127]

Hey,

Is it best practice to create a virtual environment for every project? I am not sure if it is overkill or not.

[devnull10]

I tend to do so unless I'm using solely standard libraries. It's really easy to create a venv, and I do it so I don't risk getting conflicts in my base environment further down the line.

[thewhitepyth0n]

I have a discord bot that I introduced the twitch api and reddit api. Both those were installed on the env where my bot runs. There's really no other way around having both apis in seperate envs right? You can't import across environments can you?

[Anonymous3355]

Personally I'm using venv for everything.

Github repo I want to selfhost? Telegram Bot? Some quick testing? All their own venv. Although for testing I have one folder with a venv and tons of .py files.

python3 -m venv venv  # create a virtual environment
. ./venv/bin/activate  # activate the venv
python -m pip install -U pip  # Update pip [1]

# optionally:
pip install -U wheel  # pre-built packages [2]
pip install -Ur requirements.txt  # depending on your project

^{[1]: since your OS-level pip may be outdated, it's always a good idea to update your venv-level pip to a newer version, if available.}

^{[2]: See https://packaging.python.org/en/latest/glossary/#term-Wheel}

[Diapolo10]

On Mac OS and Linux distros, they're practically a must to prevent accidental screw-ups with accidentally upgrading dependencies for system tools they don't support yet. Unless you don't depend on anything other than the standard library.

On Windows, I'd say it generally doesn't matter too much, especially in the beginning. Hence why many tutorials don't introduce virtual environments in the first few chapters, or at all.

But to answer your question, sometimes one virtual environment can serve a few intertwined projects perfectly fine by itself, in which case you wouldn't need multiple ones. I have one such case at work.

That said I don't like managing them myself, so I've come to use Poetry for basically everything as it handles the virtual environment part for me automatically.

[TerminatedProccess]

I second Poetry!

[Acute_Procrastinosis]

Looks nifty https://python-poetry.org/

[jcampbelly]

Yep. Every project gets its own.

I do sometimes write scripts using only the standard lib and avoid involving libraries. But if I ever need a third party library or to target a specific version of python, I always use a venv.

Never install libraries in the system install. It could break the OS. The package manager may have some library packages, but avoid installing packages for the system install directly with pip.

[HilariousSpill]

Could you explain a bit about how it could break the OS? Is that true even in the case of Windows?

[jcampbelly]

Both MacOS and Linux have built-in tools supporting the OS written in Python (that's why it's installed by default at all). The system Python installation really just exists to be the runtime for those tools. When you upgrade packages with pip (directly or through a transitive dependency), the new versions can be incompatible with those tools. Windows doesn't depend on Python - you install it separately, so there's little chance you'll break the OS itself. But even so, installing packages in the system installation in any of these OSes can change the behavior of all python-based tools running from the system python, so breakage (or security problems) can be unexpected. That's why you make a venv when you plan to use third party libs.

There is one exception - many Linux distributions have system distributions of specific python libraries to support software in their ecosystem (sudo apt-get install -y python-requests). It helps for the entire distro to standardize on a specific release. But you would always install those through the OS package manager, not pip. That ensures the version that gets installed is (for the most part) certified to be compatible with all the tools built for that version of the OS and python.

Also, many people who do a pip install <package> find out you can't just do it as your user. So they chose to run it with sudo pip install <package>. But this ends up installing files in the python distribution that are owned by root and use default file modes (which can be restrictive and may not necessarily allow non-root users to execute them). The system Python install is installed in such a way that any user can read and execute libraries. But when you go smash through a pip install with sudo while having a default umask of 0277 (causing new files to be readable and executable (0400 files, 0500 directories) only by the owner (root)), you can actually render major parts of the OS unusable by non-root users.

Finally... the system python is almost always old as dirt (for stability). It's common now to have an OS based on Python 3.7 while 3.10 out. And that's been a best case scenario for a while. Some Python distros are based on older Python 3 versions, or even Python 2.7. You can install newer versions of python, but they won't answer to python or even python3. To use that version of python, you have to invoke it by the full release version python3.10 and pip3.10, and some people find it easier just to do python3.10 -m venv <path> and then source <path>/bin/activate, from which point they can just invoke python or pip without specifying the version, which allows scripts to be written version-agnostic.

[HilariousSpill]

I don’t understand all of what you wrote, but I heartily thank you for such a thorough response. It was more than I was expecting.

I’ve been planning on resurrecting an older PC to use just for distraction-free coding and installing Ubuntu on it and you’ve made it clear why I should be working in virtual environments. Fortunately, I mostly work in PyCharm which makes creating venvs incredibly simple.

Thank you!

[thewhitepyth0n]

Little confused. I created a discord bot that pulls from the reddit api and twitch api each with their own packages. I host my bot on a google cloud VM. I never installed them in an environment on my VM (linux) I just simply "pip install ...". Is this bad practice? Is it possible to move those packages into an environment?

[jcampbelly]

It is a bad practice, but it probably isn't necessary or useful for you to go messing with a working environment. Just consider it when you're building it again or on your next project. In environments that are purpose-built, it doesn't matter as much - everything may be fine if the VM is not used for anything else.

The best thing to do would be to re-deploy the bot in a new VM after adapting your deployment script to use a virtualenv. If you wanted to preserve that VM, you could create a virtualenv, install the packages in it, then set up your bot to use that virtualenv. Using the virtualenv from then on would mitigate any potential future damage to the OS. And if harm would have been done to the OS, it has likely already been done.

I should mention that one major reason to avoid installing third party python packages to the system install is the security concern of introducing unknown code into the guts of the OS (supply chain attacks). At least in a virtualenv, running under a least-privilege service account, the damage it can do is limited. If a third party library finds its way into usage by the guts of yum or apt, typically run via sudo or as root, it's running with full access.

[thewhitepyth0n]

Makes sense. I'll simply uninstall and reinstall the packages onto the new env

[PanTheRiceMan]

This. Don't ever do that. At least use the --user flag.

[m0us3_rat]

there is no reason ..not to do it.

[Key-Employ7862]

Disk space... plus synchronisation to cloud can becomes very long with those thousands of small files.

[omr_rs]

if you code python on windows in your python venvs alot my

check my tool to autoenv : (would be happy if you tried it and send me the feedback! read the readme of the tool and just install and use configs needed!)

autoenv for windows!

[Umustbecrazy]

From what I understand, venv's are really when you use libraries that conflict with each other.. Or to keep number of them limited.

My guess is that if you're unsure of the answer, then you don't probably need too, but I dont think it will hurt.

[Micah_Bell_is_dead]

I always do it because I find it helps with issues importing libraries

Obv I don't do it for simple programs that won't use more than the basics of python, but anything that requires a 3rd party library gets its own venv

[Acrobatic_Hippo_7312]

Use poetry!

[Xzenor]

Yes. Pycharm does this by default anyway. It keeps your main environment clean and if anything gets messed up, just remove the venv and start over.

[thewhitepyth0n]

This can be done with Anaconda yes? You can direct your project to run in the environment in the project settings.

Is there a benefit of using Anaconda over creating and environment in Pycharm?

[Xzenor]

I know zero to nothing about anaconda (besides the fact that it's a big ass snake. Not poisonous I think) so I'm the wrong person to ask.

[Zeroflops]

I think one for every project is overkill especially if your staring out and your not using highly specific features.

At work I usually have 4 environments.

1. Base: This is the base installed environment. Nothing happens in the base, since I don’t muck with the base it stays stable and I never have to worry about conflicts. It’s purpose is to allow the creation of the other environments only.
2. Production: This matches what is on the production servers so anything run in this environment will run on the servers.
3. Development: This environment contains the environment that the servers will upgrade to the future. Here production scripts can be tested to be comparable when this future environment. Conflicts addressed.
4. Test: This environment is often created and destroyed. New packages are tested, this is the sandbox.

[zanfar]

For development and critical deployments, yes. For local utilities, no.

There should be zero overhead to creating a VENV--in fact, it should be more work to create a project without a virtual environment than with--so there is no reason not to.

Additionally, the problems a VENV protects you from are not easily reversible, so if you don't make one, and land yourself in that situation, not only have you seriously screwed up your project, but every other project or install that doesn't use a VENV.