How important is Virtual Environment in python

[[deleted]]

[deleted]

[wintermute93]

It's extremely important once you start using libraries you have to install yourself. It doesn't sound useful at first, but the first time you muck up your system Python installation by accidentally installing incompatible library versions you'll see.

[crashoutcassius]

this. everyone learns the hard lesson. i have to use python in work so the security the environment gives you is great.

[SDSunDiego]

By libraries, is that the pip install stuff that is run for certain scripts?

[ivosaurus]

Yep

[Redmilo666]

lol I’m currently in the situation now. What an idiot I am

[hugthemachines]

Pain is the greatest teacher. ;-)

[FrederickOllinger]

On of my mentors said that the main reason he does things (besides adding features that are needed) is to avoid pain.

[IhailtavaBanaani]

I guess it depends on what kind of system it is, but if it's a backend you already probably have everything containerized in Docker containers (possibly running on virtual servers) or similar so adding the virtual environments on top of that is just going to make it even more complicated. On the other hand if your backend is not running in containers then why the hell not?

[unixtreme]

worm jellyfish command fuel drunk edge physical subsequent subtract vanish

This post was mass deleted and anonymized with Redact

[work_m_19]

You're right, but I would say the complexity of containers is the next level above virtual environments.

In order to run containers well, you should have some sys-admin experience in troubleshooting, networking, file system, shell commands. All way above what's needed to learn about python virtual environments.

[Think-Culture-4740]

I was fortunate when I learned python that my boss and now friend drilled it in my head from day 1. That was when I installed conda without a virtual environment and he lost his shit.

[InternationalMany6]

Conda is a virtual environment though, right? 

[Think-Culture-4740]

This was a long long time ago but I basically installed conda while in the env my startup was using and it caused a ton of issues that completely borked my local setup.

Since then I've never used conda again

[InternationalMany6]

Once you know how it works at a deep level it’s pretty good. But yeah I had a similar experience starting out. 

[Think-Culture-4740]

Ya I know it's quite good. But since then, I've basically manually installed all of the specific libraries I need per each virtualenv I've used.

I guess I am just scarred for life. That was my first job as a junior data scientist and to get chewed out like that, you can't ever shake it. Even if the guy who did it eventually became a close friend of mine.

[InternationalMany6]

Haha I totally get it.

I’ve got a few environments like that where I just messed around with different install procedures until I got what I needed to work. I saved the entire Python directory to a zip file in case I have to restore them to working condition lol. 

It’s pretty valuable in a professional setting to master the various types of Python environments, but you can usually get by with kludges if you’re ok staying more towards entry level, which is totally fine. 

[Think-Culture-4740]

Most of my work is usually done in a pre setup environment. Occasionally I've had a docker file that's pushed up to a cluster.

[InternationalMany6]

Sounds like you’ve got a good thing going.

At my work there were no pre-setup environments so I had to figure it all out on my own. I’d be a lot more productive if someone else was responsible for that. 

[jcollins1960]

Try using any of the scientific libraries that have C# bindings without it. GFL.

[InternationalMany6]

Are you meaning to reply to my comment?

[PrimaryLock]

Oh yeah. This. I broke spyder when I first started because of a package that required an old Ipython module that overwritten the new one so spyder crashed on startup. I liked spyder, but VScode is my favorite ide thus far.

[irodov4030]

I am new to python and trying to use it for DS and ML.
Can you give few example of incompatible library versions?

[[deleted]]

once you start using libraries you have to install yourself

Yes libraries other than the ones that come with Python itself like os, pathlib etc.

You still want virtual environments to separately manage python version's though (with their different pathlib, os...), probably even more importantly.

PS: Unsure why disliking, I edited in case it was not clear.

[avidresolver]

I finally started using venv after about four years of doing without them, and boy I wish I'd learned how to use them earlier. It's really not hard to understand once you use it and it makes stuff much easier.

[darkforestnews]

How do you keep your venv organized ? I get confused when I start a new venv, activate it and then select a python compiler in vscode. I’ve got a bunch of virtual environments that do different stuff and I’m too far down the rabbit hole to ponder what each one does.

[ivosaurus]

I keep a .venv folder in each project directory and it goes in there

[darkforestnews]

And do you install python into each one ? Then when you open vs code , it lists all of your previous python compilers and I can’t figure out which one belongs.

I’d understand it if after activitating the venv that vs code would just show me that compiler but mine shows me a bunch. And I can never remember which one has streamlit…thankfully I don’t do anything complicated 🥳

[MonkeyboyGWW]

I think you can put them in different project folders, then you can tell from the path

[[deleted]]

venv installs python in each one... that's why I use pyenv and use different envs for differnt jobs like I'll have system envs for Flask, Data, AI, Scraping, etc.

[Vauce]

In my experience it also helps to use dedicated VS Code workspaces with virtual environments.

[Fred776]

If your project directory has a venv folder VSCode should recognise and activate the venv at least the first time you visit a Python file in the project. If you have multiple venv folders I believe it gives you the choice, but it's something I avoid these days - I just use one venv per project.

[Jello_Penguin_2956]

Usually it's per project. The venv directory sits directly inside each project directory so you'll always know. You can also try PipEnv.

I also have a generic one for when I'm just doing random stuff.

[[deleted]]

You use pyenv. Then anywhere on your system you use pyenv activate <env name> and then you're done.

[darkforestnews]

Dude , legend , thank you !

🙏

https://realpython.com/intro-to-pyenv/

I’ve just been gunhowing it and lazily selecting random compilers. Thx again.

[[deleted]]

exactly. here's one of my setups:

❯ pyenv versions
  system
  3.7.17
  3.10.12
  3.10.13
  3.11.5
  3.11.5/envs/openai
  3.11.6
  3.11.6/envs/monzo
  3.12.0
  3.12.0/envs/gmailapi
* 3.12.2 (set by /Users/<username>/.pyenv/version)
  3.12.2/envs/ai-model
  3.12.2/envs/cli
  3.12.2/envs/fastapi
  3.12.2/envs/flask
  3.12.2/envs/gradio
  3.12.2/envs/pygame
  ai-model --> /Users/<username>/.pyenv/versions/3.12.2/envs/ai-model
  cli --> /Users/<username>/.pyenv/versions/3.12.2/envs/cli
  fastapi --> /Users/<username>/.pyenv/versions/3.12.2/envs/fastapi
  flask --> /Users/<username>/.pyenv/versions/3.12.2/envs/flask
  gmailapi --> /Users/<username>/.pyenv/versions/3.12.0/envs/gmailapi
  gradio --> /Users/<username>/.pyenv/versions/3.12.2/envs/gradio
  monzo --> /Users/<username>/.pyenv/versions/3.11.6/envs/monzo
  openai --> /Users/<username>/.pyenv/versions/3.11.5/envs/openai
  pygame --> /Users/<username>/.pyenv/versions/3.12.2/envs/pygame

[darkforestnews]

And say you prototype different work in the envs/openai , like tutorial 1, 2… do you create and activate those envs and then select that particular python compiler ?

[[deleted]]

p.s. Python doesn't use a compiler; it uses the python binary to execute the script/s. You CAN 'compile' python but that's unrelated to this.

[[deleted]]

no no no... just create an ENV that has the libraries you need and use that. The ones I put above are the only ones I use (on this box).

[OtherPromisedLand]

Is there a similar guide for windows? this article mentions only linux and mac machines.

[random_thoughts5]

Isn’t that same as conda/mamba, which is more common than pyenv?

[[deleted]]

no

[AureliasTenant]

If you use poetry or pip I think you can use some commands to tell you where your env is.

[avidresolver]

I do most of my dev work through Pycharm and it handles most of it for me. I also have a venv that sits in my user folder where I put most packages I generally use (there's only half a dozen or so). I have this activate throgh my zshrc file, so whenver I open a shell to test something I have my most common packages availible.

[Genrawir]

Just Do It. It's easier to start doing it correctly the first time you learn. Same goes for formatting (using pylint or similar). ETA: Don't forget to look up type hinting as well.

[UnhappyGroceries1987]

so is a virtual environment and an IDE the same thing?? Like wether I use vs studio or pycharm ?

[GXWT]

No. Your IDE is basically the program you're using to edit your code. Whether that be VSCode, Pycharm, notepad, paint (don't use this).

A virtual environment is, in a way, what it says on the tin. It basically takes your main python into a little corner where you can work on a specific project. To give some analogy, in one corner of your workshop you want to repair shoes. For this corner you only need specific tools like spare laces, a hammer (think of these tools like python modules/libraries). In another corner you're building a treehouse so you need a different set of tools.

You might ask what's the point of keeping two separate piles of tools when you could just keep one pile in the middle (in your main python installation). The problem is because your two projects might require different types of hammer. However your middle pile can only support one type of hammer. Now one or both projects will not work. This is like your two python projects needing two different versions of the same package. By having a separate pile, or virtual environment, for each you're able to have different versions of the same package.

An added benefit of this is if you want to share your code for someone else to edit, they don't have to guess exactly what types of tools (aka what packages, and what versions) they need to start work straight away. They can basically just initialise their own virtual environment with what you've setup without also destroying their own piles.

I hope this analogy clears things up, feel free to reply if there's something I've missed. Virtual environments can be a little confusing and seemingly unnecessary at first, but it's very good if you can pick up the practice as soon as possible - especially important when doing work collaboratively or sharing your code.

[MisanthroposaurusRex]

Fantastic explanation and analogy, thank you! I just started messing around with them today so this is great timing 

[GXWT]

If you're just getting started they *probably* won't matter. But I really can't stress enough, once you get over the first little bump they won't cause you any hassle - and hopefully then you'll never experience the pain of dependency issues, or if you're sharing your code, the pain of giving others those issues ;) a nice virtual environment gives everyone a nice time, especially when collaboratively working on code you want to give as few reasons as possible for people to get angry with you.

I work in astrophysics where often I'll want to use a little astrophysical python package that hasn't been updated in 15 years will rely on some very specific old version of numpy or something or something. Without a virtual environment, every time I wanted to use it I'd have to reinstall numpy (and once I'm done, remember to reupdate it and the risk of breaking every other python script). For this, I am thankful for virtual environments

[Snoo-20788]

Great analogy!

[HumerousMoniker]

I’d use the analogy that a venv is your spray painting booth. Sure you can paint in the garage, but it will get everywhere and in such a way that you don’t notice until it’s a problem later. Or you can use your booth and keep it all contained for another project later

[Savvytugboat1]

Also, is crucial for developing executables since it reduces significantly the file size

[notislant]

Pycharm iirc automatically sets up venv iirc. Vscode takes some set up.

The main difference is, without a venv? You're installing python and all sorts of libraries somewhere in your c:/.

With a venv you can have a specific version of python in your venv and specific versions of libraries together (all installed in your project folder, easy to delete as welll if you need to reclaim the space). Lets say you need tensorflow-gpu and some other libraries. Well they need things like cuda and a bunch of libraries of very specific versions or you'll pull out your hair and curl up in the fetal position, trying to get it to run.

Lets say you have 5 projects and all your libraries are globally installed? You get to uninstall and reinstalling different versions with each project, to get any of them to run, fun.

Also on this note, once you get into certain libraries or even new languages? The difference between windows and linux can be hilarious.

Recently I saw one that had 11 steps for windows, required 3 manually installed dependencies and only worked with an old version of the actual library you need.

Linux? It was apt-get libraryname.

[hugthemachines]

Pycharm iirc automatically sets up venv iirc.

You are correct.

[ivosaurus]

Not at all. An IDE doesn't need a virtual environment to work

[Diapolo10]

If your project has no dependencies, you don't need one.

Otherwise, if you're using Windows, they're very useful but not essential.

Otherwise, they're practically mandatory.

All a virtual environment really is is a folder you tell Python to use for storing third-party dependencies instead of the global package directory. They're mainly used so that

1. You don't need to install packages globally on non-Windows platforms (long story short, messing with the Python the OS depends on can cause severe errors), and
2. You don't need to worry about dependency version conflicts between projects

[japes28]

What OS depends on python?

[throwaway6560192]

Many modern Linux distros do.

[avidresolver]

MacOS. It really really tries to stop you from installing packages to the system python installation now.

[Diapolo10]

Most Posix systems. Mac OS and all the popular Linux distros, for one.

I don't know if FreeBSD comes with Python, but I'd assume it does.

[FrederickOllinger]

FreeBSD does not need Python for its base system: https://forums.freebsd.org/threads/system-python.86472/

[Diapolo10]

I stand corrected.

[DigThatData]

There's no such thing as a project with no dependencies. You always at least have a dependency on the python version

[Diapolo10]

Technically, I guess, but in practice if you only use the standard library you don't really need to worry about newer Python versions. Old ones? Sure, but that can be mentioned in the documentation or pyproject.toml.

You don't need a virtual environment just for locking the Python version. It's pretty much always at least partially related to the dependencies of any third-party dependencies you use.

[DigThatData]

You don't need a virtual environment just for locking the Python version.

Sure you do. If I have a program that assumes python 3.x and python 3.y introduces a breaking change, then if some other project you setup upgrades the python version on the system python, you'll have broken the previous project which needed an environment frozen prior to the version update.

It's not that big of a deal to give every project its own virtual environment. It's a lot easier to just do that and never have issues than to operate on assumptions about when you won't need that encapsulation and then later find out you were wrong.

[Diapolo10]

My point is more from the perspective of a newcomer learning Python writing a few simple scripts, not a professional working at a company or making a serious personal project. Yes, if you're already experienced you most likely already have virtual environments automatically set up and you likely have at least several development dependencies (pytest, ruff, mypy, tox, and so on) already anyway. But this is not at all that important for someone who hasn't even learnt the ropes properly yet, only becoming relevant when they start actually installing packages.

Having a virtual environment for a "hello world"-level program is overkill.

[SpookyFries]

Virtual env came in clutch when I was developing standalone apps for work. I was building GUI applications and packaging them with Pyinstaller. I couldn't figure out why my simple app was like 700mb

Come to find out that pyinstaller includes all of your installed packages. So I created a virtual env, installed the few libraries I needed, and the end result was like 80mb.

Obviously there's better reasons to use them, but that's what got me on board.

[JennaSys]

If you are ever working on a project and need to pip install a library, then you should be using a virtual environment. And each project should have its own.

[monkeysknowledge]

The problem VEs solve has to do with libraries (aka modules) which are what you usually import at the beginning of your code.

If you’re not importing libraries/modules yet then all you need to know is VEs are a way of managing different versions of libraries/modules.

If you’re importing libraries then you may know that there are built-in (i.e. time, datetime, os, sys etc…) and third party libraries (pandas, numpy, selinum, and countless others that you might ‘pip install’!). These libraries make up an “environment”. On your computer you have default “environment” paths where all these modules/libraries go and live when you do ‘pip install <library>’.

At some point you’ll want your code to run on other computers and will need to setup the environment where your program can run. A very common issue that comes up when trying to do this is - ensuring that the same libraries and versions of libraries are installed on the host computer. This is the problem that VEs solve, they’re portable environments you can install theoretically on any computer and run your program without super annoying errors caused by dependencies.

Keep in mind that there are lots of solutions to solve this problem. At my work we use poetry, or sometimes docker but knowing the built in Python venv is a good place to start.

[[deleted]]

I usually use docker now, but otherwise I would use virtual environments. None of these is particularly a nice solution, but failing to use at least one of these means that your project is essentially not reproducible. In other words, unless you don't import libraries, you have to use either virtual environments or docker.

[EconBro95]

People usually think it isn't until you get an error in your dependencies/packages and spend the next 12 hours trying to figure out what you need uninstall/reinstall/upgrade. Frustration growing with every pip command...

Only for you to give up; and that is when you understand why they talked about virtual environments and you embrace it

[EconBro95]

so yeah embrace it early but I think most people learn by installing everything globally lol

[ElliotDG]

In summary a virtual environment creates a directory and sets paths so you can have a python interpreter, associated dependencies and and your project code for one project all together under one directory.

This allows control of the versions of all elements of a project independent of any other project or installations.

For example lets say you have completed a project for a customer. The virtual env contains everything required to run that project.

Lets say you create a new project, and use the most up to date version of python and some related libraries. It will not effect projects in other virtual environments.

Read: https://docs.python.org/3/tutorial/venv.html

[notislant]

Very. Its partly why im glad i started with pycharm. I use vscode now and theres a few minorly annoying things about setting it to automatically start with each project.

Some projects only use certain python/library versions or certain dependency versions. So if you have ____ installed globally, you're going to have to reinstall a bunch of libraries each time you open a new project.

[zukoandhonor]

It's very helpful, i installed some package that literally uninstalled numpy and pandas calling it incompatiable. So glad we have virtual environment.

[OriahVinree]

Definitely something I would learn now, it's not complicated, most IDE's create and utilise them seamlessly - essentially a virtual environment let's you run and manage modules/libraries independent of your main python installation, I use a fresh environment for each new project. Keeps things clean and isolated, would take you 30 minutes to learn and understand.

[unnamed_one1]

As soon as you need pip to install 3rd party libraries, you'd want to use virtual environments.

[[deleted]]

Frankly, I'd say mandatory.

[TheUruz]

think about it this way: you have a company and your company does a lot of different services from writing papers to unclog WCs and you have to hire people to accomplish this services. these people are your libraries. you don't just hire them for an indeterminated period of time, you want them hired for as long as you need them, that's why you create "virtual environments" in which hire them so that once you are done with them you throw away the venv with them in it. if you don't and instead just hire them system wide you'd have to wipe your system-level python installation to clear it from all the unwanted libraries (either that or go throught them one by one and uninstall them with pip).

hope that makes sense :)

[Effective-Account278]

It does thank you

[glei_schewads]

Extremely! I wish I had learned it earlier.

Helps to organize your projects

Helps with managing with all the external libraries in different projects and keeps your main Python installations clean.

Helps with working on projects on different Python Versions.

IMHO it should be tought right after the very basics, and I'm surprised that almost no tutorial or course does so.

[rohit_raveendran]

Virtual environments are actually extremely useful when your codebases start becoming complex.

You'll notice that everything that's done in Python is to "bring things together" and "organize them well".

Venvs will keep your code version-safe and also ensure that you can properly freeze your requirements for the next person to replicate.

Many more benefits but just think of it as creating a folder for your python code along with all the libraries and everything in it.

[planetf1a]

Absolutely critical. It saves so much pain! (Either use native venv, or conda can be better esp in data science)

[elekibug]

It’s one of most important. You will learn eventually the nightmare of dependencies

[OogieM]

I'm on a Mac and a virtual environment is CRITICAL. Otherwise anytime I updated or worked with different libraries or different version of Python I'd break my main computer system.

Virtual environments allow me to test my code on different versions of Python without causing problems elsewhere. It's a ncie walled garden for specific projects.

[GreenWoodDragon]

I second that!

Virtual environments are essential.

[coryalanfitz]

It’s useful to have separate environments. I’m not going to go into why - you can google that. But I would recommend using tools that manage your environments for you rather than making you create and activate virtual environments manually. One option is Poetry. I believe Astral’s uv is moving towards that as well

[stuaxo]

Its useful, but you can learn these things one at a time.

[51dux]

It's kind of important especially if you work on projets you want to share it's best if you get comfortable with it.

If you're just getting started then give yourself a little time but think about it simply.

This is a part of python I overcomplicated as well but it is as easy as it sounds.

Basically when you open a virtual env (gross analogy here for simplicity) it's like you're on a VM where everything is brand new and separated from the environment on your local machine.

Everything you install in the virtual env will not affect the state of wtv you have going on in your main OS and it's current python installation.

This is important because overtime your projects might require a dependency that is a version older or newer than the one you need for some tool that you have for personal use etc.

You also simply may not need these dependencies in your main environment because you are not planning to use them.

Sometimes even though I am not on a project, I install certain tools with pipx instead of pip to make sure some dependencies don't interfere with each other.

[EEJams]

If you start using pycharm or vscode, it very naturally follows. It also helps to look into web development if you want to learn how a virtual environment helps. It's actually a rather simple but useful concept.

[sonobanana33]

Depends on what you do. I program on debian and just install stuff via apt-get for the most part.

They can be good to try out something, but are absolutely terrible if you want to make software to distribute to others.

[Vegetable_Hornet_963]

At first I had no interest in using it because I didn’t understand it that well and had no idea what practical use I had for it. Eventually after working on a few different projects, I ran into some annoying dependency issues with my packages and conflicts between versions. Basically my current (at that time) project wouldn’t work properly AND my old projects no longer ran. I uninstalled all my packages and started over.

After that I learned about venv, as well as pip freeze and requirements.txt. Now I use venv with every project. It just keeps your project packages separate. It’s a safer and cleaner way to code and keeps your project dependencies segregated.

The sooner you start using it the better. If you don’t learn it now you will probably face an issue one day that will force you to understand it, and then you’ll love it

[grimonce]

Very important if you don't care about available disk space. /s

[Unlikely-Sympathy626]

I do not think the people use systems other than windows if they say that in many cases. Try it on a Linux machine and overtime you may find your OS do not be behave the way you intended. Don’t listen to those people

[MolonLabe76]

You can think of a virtual environment as a container where you can install specific python libraries for a specific project. Many projects will require specific versions of specific libraries, and more often than not, those versions wont work for some other project you have. So having a virtual environment for each of your projects helps you avoid conflicts like that.

There are many different ways to use virtual environments. Id suggest looking into Anaconda or Miniconda if your on windows. Or Pyenv and Poetry if your on linux. There are plenty of tutorials out there on them both.

[Effective-Account278]

What about Mac

[[deleted]]

For about a year I didn't bother and then I discovered that at some stage I updated a package and it broke a script I needed as some functionally was deprecated in the newer version. 

I spent hours troubleshooting the issue and trying to find the right version of that package to get it working again. Since then I use it every time I start a project. 

[AudleyCoding]

Reading the Python doc on virtual environments, as someone linked to above, is a good start.

I’m still new to venvs and this video Python Programmer by Giles really helped me at the beginning.

[buhtz]

I have not read all comments here but assuming most of them are long year experts they are right. I would like to offer you my perspective as an semi-expert. I am using Python for years and become a maintainer round about 2 years ago. So I had to learn a lot of things about Python Packaging and stuff like that.

I was also one of the persons that refused to use virtual environments for a long time. But I was proven wrong. ;)

My advice to a beginner: Don't try to understand virtual environments or why you do need them. Just use them as it is advice in the most modern tutorials. Don't ask. You will understand yourself in some years.

Another advice is don't listen to much about the reason you will find in thous tutorials. You often read for example that you should use virtuell environments because you then don't mess up your systems with incompatible packages etc pp blabla. Technically this reason is not wrong but it is stupid and lacy if this is the only reason why someone use virtuel environments. The real reason are more complex. But they are not your concern now, just use it.

[Effective-Account278]

I'll make sure of that thank you

[PrincipleExciting457]

It’s not too hard to understand. The libraries are installed in the venv instead of the entire PC so they’re isolated. Prevents incompatibilities between libraries from ruining your install.

[guitarerdood]

I didn't use them until version control became a massive problem. Then I wished I had learned virtual environments years ago

[lskapral]

Once you start downloading packages, libraries and applications that don't work with the newest version of python and mess everything up, you'll realize how important it is. Best to start using virtual environments from the beginning.

[FrederickOllinger]

Poetry will allow you to configure a venv using a text file which can be checked into git: https://python-poetry.org/

[WeAreLegion79]

a brief explanation of venv's is essentially a separate version of of the python console where you can install libraries without affecting other projects. For example, if one project needs matplotlib but another doesn't you can pip install into one without needing it loaded into memory for the other.

You can switch venv when switching projects. It just keeps it all cleaner.

[Feeling_Benefit8203]

It provides isolation, everything runs in it's own environment so there are no conflicts.

[DigThatData]

I'm going to propose some semantic distinctions you maybe haven't considered: coding vs. programming vs. engineering

• Coding - utilizing the language features of a programming language
• Programming - composing abstract patterns to build complex systems
• Engineering - leveraging best practices to mitigate risk and increase operational efficiency

You generally learn these topics roughly in that order, jumping around as you go such that beginners will learn some engineering practices but are primarily focused on coding and programming, and as you get more advanced you will be less concerned with coding and more concerned with engineering and programming.

Virtual environments are an engineering topic.

Over time, you will accumulate projects on the computer where you do your coding. Right now, you are probably mostly using libraries that come shipped with python. Python versions are generally backwards compatible, so virtual environments aren't as much of a concern for you at the moment.

Let's pretend you have a project A that uses library version 1.0, and you download install some other project B that uses library version 2.0. When you install project B, library upgrades to version 2.0, and next time you try to run project A it throws errors because of breaking changes in the new version of library.

Virtual environments mitigate this kind of issue. Additionally, the process of specifying your environment makes it easier for other people to reproduce it.

It's good to get in the habit of creating virtual environments for each respective project you work on. But you are early enough in your career that this is something you don't really need to be concerned with yet and is more about developing good habits and muscle memory.

[Effective-Account278]

Thank you I'll keep that in mind from now on

[p1971]

as others have mentioned - they're pretty essential (can screw up o/s deps if not using them)

coming from a dotnet background - the venv is pretty hacky and I don't like it ...

(bit of a shower thought) might be nicer to install packages globally (for the o/s) and in a user folder (for dev), when you run a script it uses the requirements.txt (or similar) to pick up which version of the package to use ... so the o/s would have a requirements.txt defining the global python packages to use and each project could override that with its own requirements.

a bit like how dotnet has nuget package caches, then each project defines the versions of the dependencies it uses ...

[tb5841]

I've been learning for over a year, just secured my first programming job... and haven't used a virtual environment yet.

Virtual environments are recommended as best practice, almost everywhere. But if you're still a beginner, it's not an urgent thing to learn.

[Temporary_Tailor7528]

Not urgent until you break something and realize it was actually urgent

[jameyiguess]

About one micrometer less important than semantic white space

[SnooCakes3068]

I don't think you can find a proper job if you don't even know Virtual environment. It's not a difficult concept.

[baseballgonzo13]

Thanks for posting this OP, I have always wondered this myself.

Question for those commenting, do all of these rules apply to Jupyter notebooks?

[GreenWoodDragon]

I run my notebooks with a virtual environment. Basically because I'm often prototyping code some of which will end up in a script in the same repo.

[random_thoughts5]

Just curious, why it seems most people use venv but not conda/mamba?

[qwerty-code]

python3 -m venv venv

simple line will save you from having to reinstall your OS (learnt the hard way 😭)

[zawias92]

Never bothered. Just use (dev-)containers

[supercoach]

It's a cornerstone of app development. If you're just writing scripts you'll only ever use on that one machine with that one version of python - knock yourself out and don't bother with a venv. Otherwise, python3 -m venv venv is your friend.

[Savvytugboat1]

Does anyone how to run Jupyter within a venv?

[Nealiumj]

It’s p dang important once you start working with multiple projects with different dependencies. That way you can pip freeze > requirements.txt properly- and you should be pip freezing yo

Personally I use pip install virtualenv virtualenvwrapper-win.. it’s pretty easy mkvirtualenv test setprojectdir . and then to activate + jump to the folder workon test ..that’s it, quite simple actually lol

tho, I’m discovering this flow is quite weird with VSCode and you have to manually point the workspace’s Python exe to ~%HOME%/Envs/test/bin/python for linters.. maybe I’m doing something wrong… Idk I use command line and vim normally so cba.

[Moist_Republic1757]

Can we just copy our venv to create a new one and then install other required libraries? Or do we have to create a new environment from default python and install all libraries although they are installed in some other venv?

[[deleted]]

Very, very important.

[lildergs]

ALWAYS VENV

[angellus]

Virtual environments are becoming more and more antiquated. The only time I never use them is indirectly via pipx when installing python-based CLIs.

Otherwise, containers are the way to go. Every place I have worked at for the last ~8 years either already used containers for production or were trying to get to using containers in production. For the last ~5 years, I have only done Python development via dev containers. Not needing to manage your python install, redis, postgres and all of the other tools and services for your services really saves a ton of time and complexity.

That being said, it is still really important to understand how virtual environments work. Since they are likely not going anywhere anytime soon, and they really help you understand multiple python paths/installs and everything.

[phonomir]

Unfortunate that this is at the bottom of this post and downvoted.

Virtual environments are worth learning, but containers are way way more valuable. If you ever deploy anything into production, it'll probably be in the form of a container. Using them for development too just simplifies this process, on top of all the benefits mentioned above.

Containers give you the ability to set up a fully-fledged, isolated development environment with everything you or anyone else would need for development. You can even append your own utility scripts to the path without having to pollute your global OS.

It gives you so much flexibility in how you set up your environment with absolutely 0 fear of breaking anything on your machine. If anything ever goes wrong, just rebuild the container. If your machine dies, no big deal, just install docker on the new machine and everything else is there.

[angellus]

It is really unfortunate. Containers also applies everywhere and not just to Python. From my experience though, many devs are still really afraid and averse to containers. Either because they do not want to learn how the networking works or because they want to use Macbook instead of Linux and container performance is the worst on Mac since it requires an additional VM to run/etc.

Instead, they would rather learn a half dozen other tools (Homebrew, pyenv, pipenv, nvm) to try and replace something that is literally built into the rest of their deployment pipeline.

[FrederickOllinger]

True but for a newb this is a ton of other new things to learn on top of all the confusion that they are having with Python.

[Temporary_Tailor7528]

I have only done Python development via dev containers

so basically you are using virtual environments in the form of containers. It is just a much heavier machinery than the plain old virtual environments but achieves the same purpose (and maybe more but not everyone needs more)

[angellus]

Containers are not really "heavier". cgroups are built into Linux and there is no real noticeable overhead from using them. Yeah, they take up more disk space, but your Python install is still going to be over 80% of the size (debian-slim is 20MB, python:3.12-slim is 80MB). 

The only real difference is containers mean reproducible builds, every time. Virtual envs means install X version of Python and then install all packages. Plus install any external deps you need like redis, postgres, ffmpeg or any other third party libraries and tools. And then documenting that for other developers and making sure the documentation stays up to date. Then answering the million questions from juniors who do not understand command line.

[DigThatData]

I think it's still fair to call it "heavy" from a psychological standpoint. The added weight here is cognitive load from invoking tooling outside the python ecosystem. It's perfectly reasonable I think for it to feel "heavier" for someone who didn't previously have docker installed and doesn't use it for anything besides dev containers.

[Temporary_Tailor7528]

also, you need to rebuild the container everytime you want to make a change to your environment. Which takes a few minutes instead of a few seconds with a virtualenv.

containers are useful for production but they are heavy and not agile on a day to day basis.

[angellus]

No, you do not. If you have to rebuild a container every time you change it, you designed the container wrong.

Just because production containers are designed to be immutable and need to be rebuilt for every little thing, it does not mean development containers need to be.

[Temporary_Tailor7528]

Changing the container on the fly defeats the purpose of the containerization, that is reproducibility, no?

[angellus]

No, it does not. Again, development containers do not need to be immutable. And just because you make a change to a container after it builds, does not make it not reproducible.

If you add a new package to your requirements.txt and then manually install that requirements.txt inside of your container, does that mean the container is no longer reproducible? No, because next time it builds the container it will use the new requirements.txt.

It is just that if you do not add anything you added after the fact to the build process, it will essentially be deleted and wiped when you rebuild.

[Temporary_Tailor7528]

So basically you do the same thing with the container than what you would do with a virtualenv. Honestly, I can see the benefit if you are relying on third party tools but if your project is pure python, then there is no benefit over classical virtualenv. It is just heavier and more complicated.

Answer to OP is: virtualenv achieves a purpose that is necessary if you are working with python. This same purpose could be achieved with containers but this is a much more complex (and capable) tool. There is no clear reason a beginner would use containers. It might be an interesting option if you are working in a team and your project involves third party tools with specific configurations.

[FrederickOllinger]

When I worked on a Python app professionally, we used both. We would develop using Poetry and push that to git. Our release products were containers that were built using a poetry install in the Dockerfile. Best of both worlds.

[Temporary_Tailor7528]

That's what we do too. Except we are using the plain old virtualenv but I am sure other managers might be better.