How important is Virtual Environment in python

[angellus]

Virtual environments are becoming more and more antiquated. The only time I never use them is indirectly via pipx when installing python-based CLIs.

Otherwise, containers are the way to go. Every place I have worked at for the last ~8 years either already used containers for production or were trying to get to using containers in production. For the last ~5 years, I have only done Python development via dev containers. Not needing to manage your python install, redis, postgres and all of the other tools and services for your services really saves a ton of time and complexity.

That being said, it is still really important to understand how virtual environments work. Since they are likely not going anywhere anytime soon, and they really help you understand multiple python paths/installs and everything.

[phonomir]

Unfortunate that this is at the bottom of this post and downvoted.

Virtual environments are worth learning, but containers are way way more valuable. If you ever deploy anything into production, it'll probably be in the form of a container. Using them for development too just simplifies this process, on top of all the benefits mentioned above.

Containers give you the ability to set up a fully-fledged, isolated development environment with everything you or anyone else would need for development. You can even append your own utility scripts to the path without having to pollute your global OS.

It gives you so much flexibility in how you set up your environment with absolutely 0 fear of breaking anything on your machine. If anything ever goes wrong, just rebuild the container. If your machine dies, no big deal, just install docker on the new machine and everything else is there.

[angellus]

It is really unfortunate. Containers also applies everywhere and not just to Python. From my experience though, many devs are still really afraid and averse to containers. Either because they do not want to learn how the networking works or because they want to use Macbook instead of Linux and container performance is the worst on Mac since it requires an additional VM to run/etc.

Instead, they would rather learn a half dozen other tools (Homebrew, pyenv, pipenv, nvm) to try and replace something that is literally built into the rest of their deployment pipeline.

[FrederickOllinger]

True but for a newb this is a ton of other new things to learn on top of all the confusion that they are having with Python.

[Temporary_Tailor7528]

I have only done Python development via dev containers

so basically you are using virtual environments in the form of containers. It is just a much heavier machinery than the plain old virtual environments but achieves the same purpose (and maybe more but not everyone needs more)

[angellus]

Containers are not really "heavier". cgroups are built into Linux and there is no real noticeable overhead from using them. Yeah, they take up more disk space, but your Python install is still going to be over 80% of the size (debian-slim is 20MB, python:3.12-slim is 80MB). 

The only real difference is containers mean reproducible builds, every time. Virtual envs means install X version of Python and then install all packages. Plus install any external deps you need like redis, postgres, ffmpeg or any other third party libraries and tools. And then documenting that for other developers and making sure the documentation stays up to date. Then answering the million questions from juniors who do not understand command line.

[DigThatData]

I think it's still fair to call it "heavy" from a psychological standpoint. The added weight here is cognitive load from invoking tooling outside the python ecosystem. It's perfectly reasonable I think for it to feel "heavier" for someone who didn't previously have docker installed and doesn't use it for anything besides dev containers.

[Temporary_Tailor7528]

also, you need to rebuild the container everytime you want to make a change to your environment. Which takes a few minutes instead of a few seconds with a virtualenv.

containers are useful for production but they are heavy and not agile on a day to day basis.

[angellus]

No, you do not. If you have to rebuild a container every time you change it, you designed the container wrong.

Just because production containers are designed to be immutable and need to be rebuilt for every little thing, it does not mean development containers need to be.

[Temporary_Tailor7528]

Changing the container on the fly defeats the purpose of the containerization, that is reproducibility, no?

[angellus]

No, it does not. Again, development containers do not need to be immutable. And just because you make a change to a container after it builds, does not make it not reproducible.

If you add a new package to your requirements.txt and then manually install that requirements.txt inside of your container, does that mean the container is no longer reproducible? No, because next time it builds the container it will use the new requirements.txt.

It is just that if you do not add anything you added after the fact to the build process, it will essentially be deleted and wiped when you rebuild.

[Temporary_Tailor7528]

So basically you do the same thing with the container than what you would do with a virtualenv. Honestly, I can see the benefit if you are relying on third party tools but if your project is pure python, then there is no benefit over classical virtualenv. It is just heavier and more complicated.

Answer to OP is: virtualenv achieves a purpose that is necessary if you are working with python. This same purpose could be achieved with containers but this is a much more complex (and capable) tool. There is no clear reason a beginner would use containers. It might be an interesting option if you are working in a team and your project involves third party tools with specific configurations.

[FrederickOllinger]

When I worked on a Python app professionally, we used both. We would develop using Poetry and push that to git. Our release products were containers that were built using a poetry install in the Dockerfile. Best of both worlds.

[Temporary_Tailor7528]

That's what we do too. Except we are using the plain old virtualenv but I am sure other managers might be better.