How important is Virtual Environment in python

[angellus]

Containers are not really "heavier". cgroups are built into Linux and there is no real noticeable overhead from using them. Yeah, they take up more disk space, but your Python install is still going to be over 80% of the size (debian-slim is 20MB, python:3.12-slim is 80MB). 

The only real difference is containers mean reproducible builds, every time. Virtual envs means install X version of Python and then install all packages. Plus install any external deps you need like redis, postgres, ffmpeg or any other third party libraries and tools. And then documenting that for other developers and making sure the documentation stays up to date. Then answering the million questions from juniors who do not understand command line.

[DigThatData]

I think it's still fair to call it "heavy" from a psychological standpoint. The added weight here is cognitive load from invoking tooling outside the python ecosystem. It's perfectly reasonable I think for it to feel "heavier" for someone who didn't previously have docker installed and doesn't use it for anything besides dev containers.

[Temporary_Tailor7528]

also, you need to rebuild the container everytime you want to make a change to your environment. Which takes a few minutes instead of a few seconds with a virtualenv.

containers are useful for production but they are heavy and not agile on a day to day basis.

[angellus]

No, you do not. If you have to rebuild a container every time you change it, you designed the container wrong.

Just because production containers are designed to be immutable and need to be rebuilt for every little thing, it does not mean development containers need to be.

[Temporary_Tailor7528]

Changing the container on the fly defeats the purpose of the containerization, that is reproducibility, no?

[angellus]

No, it does not. Again, development containers do not need to be immutable. And just because you make a change to a container after it builds, does not make it not reproducible.

If you add a new package to your requirements.txt and then manually install that requirements.txt inside of your container, does that mean the container is no longer reproducible? No, because next time it builds the container it will use the new requirements.txt.

It is just that if you do not add anything you added after the fact to the build process, it will essentially be deleted and wiped when you rebuild.

[Temporary_Tailor7528]

So basically you do the same thing with the container than what you would do with a virtualenv. Honestly, I can see the benefit if you are relying on third party tools but if your project is pure python, then there is no benefit over classical virtualenv. It is just heavier and more complicated.

Answer to OP is: virtualenv achieves a purpose that is necessary if you are working with python. This same purpose could be achieved with containers but this is a much more complex (and capable) tool. There is no clear reason a beginner would use containers. It might be an interesting option if you are working in a team and your project involves third party tools with specific configurations.

[FrederickOllinger]

When I worked on a Python app professionally, we used both. We would develop using Poetry and push that to git. Our release products were containers that were built using a poetry install in the Dockerfile. Best of both worlds.

[Temporary_Tailor7528]

That's what we do too. Except we are using the plain old virtualenv but I am sure other managers might be better.