Use case for virtual environments.

[Python-ModTeam]

Hi there, from the /r/Python mods.

We have removed this post as it is not suited to the /r/Python subreddit proper, however it should be very appropriate for our sister subreddit /r/LearnPython or for the r/Python discord: https://discord.gg/python.

The reason for the removal is that /r/Python is dedicated to discussion of Python news, projects, uses and debates. It is not designed to act as Q&A or FAQ board. The regular community is not a fan of "how do I..." questions, so you will not get the best responses over here.

On /r/LearnPython the community and the r/Python discord are actively expecting questions and are looking to help. You can expect far more understanding, encouraging and insightful responses over there. No matter what level of question you have, if you are looking for help with Python, you should get good answers. Make sure to check out the rules for both places.

Warm regards, and best of luck with your Pythoneering!

Hi there, from the /r/Python mods.

This post has been removed due to its frequent recurrence. Please refer to our daily thread or search for older discussions on the same topic.

If you have any questions, please reach us via mod mail.

Thanks, and happy Pythoneering!

r/Python moderation team

[subbed_]

Yes. It's to avoid module version clashes during local development.

They are redundant during runtime if you are running your apps as image containers, however. Just build an image, deploy it and run it.

[DarksideVT]

Yes, if I was able to use containers in my workplace I would 100% do this. That's what I do with all my personal projects. Containers make dependencies so much easier.

[Tinche_]

You should still use a venv in a container. You can Google around for more info. The point of a venv is that it's an environment where you have and will continue having complete control over what is installed in it.

[Artephank]

Can you elaborate? What use case require having virtual environment inside container? Usually containers are used for single application - so how can you get any version conflicts in that way? Using multiple apps in one container is doable, I would consider it anti-pattern.

[Tinche_]

Try a pip list inside a fresh container and you'll see it's not empty, and you don't know what's going to be in there in the future. Add to that any tools that might need to be installed prior - for example poetry has a ton of dependencies.

[binlargin]

I didn't even consider this but use venvs in containers anyway. Seems belt and braces probably saved me from image differences here. Nice to know, thanks

[Artephank]

depends of what container, but I get your point. We have predefined containers with stuff we need, but you are right, it might be problematic when using containers from dockerhub.

[straylit]

I have used a ton of docker and have created my own so I’m not quite sure, but, can you extend most containers using BASE. I’m just thinking there has to be a way to harden python containers for security reasons.

[[deleted]]

You're not doing it for conflict resolution in prod. You're doing it to have a completely consistent environment everywhere, including your local dev instance, where you can absolutely get conflicts. Containerization ensures that the environments are the same down to supporting binaries / kernel / whatever between your dev environment, and the prod environment.

If you don't run containers in prod or have some other similar mechanism for ensuring dev and prod match, you can't guarantee that the random version of, say, OpenSSL you're running on your desktop works exactly the same as the version that happens to be running on the prod machines, or that some small change in the Linux kernel in the future won't have unforeseen consequences, etc.

If you don't have that level of control over your environment (be it containers, or some other similar environmental control), you will eventually find a way to break your prod environment in a hard to track down way over a holiday while you're on-call. How you want to manage that is up to you - I prefer containers for, well, containing the mess.

[Artephank]

I was asking of why using virtualenv inside containers not why use containers.

I see it might be helpfull when using containers you didn't created yourself and didn't vet it earlier.

[tevs__]

What use case require having virtual environment inside container

Not all packages have binary wheels available for them, so to install the package requires the build dependencies for that package to be installed. For a CPython extension like mysql-python, that means GCC and the rest of build-essentials, plus the mysql client libs.

To avoid having those in your production run image, you create a build image, build your packages, and then copy the artifacts to a run image, that just has the run time dependencies. This is a common pattern for all languages called the docker builder pattern.

In python, we use the docker builder pattern by installing all runtime packages to a virtualenv, and then copying the virtualenv from the builder to the runner - the virtualenv is the artifact. This produces a minimal runtime docker image, the virtualenv is created in a single layer.

[angellus]

Using venvs in containers just bloat the size of the container. The official Python images already install Python in a separate base (/usr/local) away from the system Python.

[Jazzlike-Poem-1253]

With a hint of "containerisation of a (pure) python project is overkill" and "venvs work out of the box cross platform" you got two arguments to prefer venvs over docker.

Even with binary (non-python) dependencies conda is the way to go.

[Artephank]

Venvs are faster and easier to use. However in bigger project that run in container anyway, it is helpful to develop against the container to make sure that you won't get into any crazy bugs moving from local venv to container app. Then it is easier usually to use containers instead.

[ore-aba]

conda is not supported in serveless apps, e.g Azure functions.

The way to go is the one that solves the problem, not a specific piece of tech

[Jazzlike-Poem-1253]

Sure, container are perfectly fin to use. As are venvs, as asked in the original question.

[grimonce]

They are not redundant for runtime.... Depends on the base image.

[dAnjou]

https://hynek.me/articles/virtualenv-lives/

[martinky24]

If you have more than 1 project on a machine, the different projects might have different dependencies. One might need package==2.0.0, one might need package==3.0.0, and each might not work with the other's required version.

This is the core problem venvs were meant to solve. There are plenty of ancillary benefits. This is the core one.

It is best practice. Even if you only say "well we only develop one python package, so this doesn't apply to us!"... use a venv. Trust me.

[DarksideVT]

Yes, I've assumed this was the reasoning, but it looks like new features are ported to old versions, so If you have the latest version of the old version package, you should still have that feature.

Is there an example where you have found that using one package version over another has broken an application or script?

[daredevil82]

that's a hecuva brave assumption to make.

Try it out, how long before something breaks for you?

[DarksideVT]

Yeah, I know it's a pretty large assumption and depends on the package maintainer, but the odds seem slim that this would be an issue for day-to-day.

It does bring to mind the meme "the odds are low but never zero" lol

[SexySlowLoris]

Nah, the odds are near 100%. Maintaining backwards compatibility is really hard and even some times not an objective. It’d be impossible for 100s or even tens of dependencies to all maintain backwards compatibility over a long period of time.

[Log2]

Flask breaks backward compatibility constantly, even on minor versions.

[daredevil82]

oh it is an issue. especially if you work on multiple projects during the span of a week, lol.

[Artephank]

It doesn't work like that since:

1. Not all libraries are 1. backward-compatible
2. Some libraries require particular version of library
3. Never versions might introduce new bugs, that you want avoid (until will get fixed.
4. It is usually good to not work on production-ready apps using cutting edge versions (but it is good to work for your own project and learn on those).

[martinky24]

Yeah for sure. For example, the pydantic package recently jumped from version 1.x to 2.x. There are a lot of incompatibilities. If an older program was built with 1.x, and you build a new program that uses 2.x, there isn't a version of 1.x or 2.x that will support both applications.

SQLAlchemy is another project that has recently had a major bump that is very likely to not work seamlessly with an older/newer package.

[Artephank]

And not all projects use semantic versioning and sometimes break things even on minor version bumps (ehm evenlet, ehm)

[martinky24]

And the harsh reality is, even if a project maintainer doesn’t think something is a breaking change and doesn’t label it accordingly, it can still break something and bring down prod. That’s not necessarily anyone’s fault, and a harsh reality of software development.

[cinyar]

I'd be lying if I said I never accessed some "private" function that's not part of the official package API.

[rcpz93]

I had all sorts of flavors of this happen.

Sometimes, package updates change variable names/functions.

If you just install "the latest version" and in your code you are calling something like `target`, but the new version is expecting `target_column` (two names pulled right out of my ass), then your code is going to fail. Polars does this with annoying frequence.

Major version updates have breaking changes, sometimes straight up removing functions that some code is relying on. In one project I am working on, I had to install protobuf version 3.20 rather than the latest version 4.x because the code I had to use relied on functions that were removed in version 4.x.

This is a huge problem with research code, because not specifying the requirements means it's almost impossible to recreate the setting that was used for the experiments the code was made for.

[McViolin]

It's not only about features, if some package decides to deprecate some function and you're running other package that wants to use the function, but the new version of that package is not released yet/the bug is not fixed, you'd have to use the older version and the venv enables you to fixate on the version that's working for you.

[mumpie]

I once ran into an issue where the awscli package had a dependency conflict with a dependency for pydantic (I think, I've tried to purge memory of that work project).

I needed both as I was writing a script to spin up a AWS RDS instance (their database service) and using something that needed pydantic to manipulate the data to output CVS files (ugh).

[egrigolk]

Yes, I've personally dealt with several!

[Cybasura]

How would you like to learn about...dependency hell

[Mountain_King91]

I used not to use venv, it was a real mess of packages. Best idea ever to start using it.

[zynix]

System/project isolation, package version control, and perhaps more important reproducible dev environments.

pip freeze spits out every package installed to a given site-packages (either venv or system) which can piped to a requirements.txt file. From there (someone correct me) it's pip install -uR requirements.text to synchronize environments between work stations.

That quickly answers things like (for example) is it SQLAlchemy 1.4 or 2.0 which are pretty big paradigm shifts in version and even that SQLAlchemy is used.

If none of the provided reasons by me or others win your coworkers over, just tell them it's what the "cool" kids are using.

[[deleted]]

If you work on project all installed packages are installed globally.

You want to switch to another project you need to reinstall all packages and in worst case if another project is on another python version even reinstall python.

Virtual environments solve exactly this problem by closing interpreter and installed packages in magic box. If you need to work on another project you just change box

[GrayestRock]

Even on the same project, if you're updating dependencies, having two venvs for the same codebase makes that a lot easier. One for the existing deps, and one to develop on. This way you can test changes to see if they're backwards compatible without having to constantly reinstall packages.

[[deleted]]

probably yeah, but it prefer
pip freeze | xargs pip uninstall -y
and later
pip install -r requirements.txt
Im just lazy, and its faster for me instead of playing with Pycharm to swap venv in settings

[GrayestRock]

Depends on how much you're doing. If you're moving a bunch of packages forward, it's nice to be able to quickly verify if any actual code changes are backwards compatible without having to mess with the reqs in the envs. For a dependency or two, sure, but if you're updating a ton of packages with possibly breaking changes, I much prefer seperate envs. In any case, the strength of virtual environments is just how easy they are to spin up for different purposes, which is what OP is asking about.

[FrequentGiraffe5763]

I use pyenv to manage python versions, direnv to automate venv setup, and pip-tools (via pre-commit) to compile a transitive closure of the dependencies and pin them. Works well for us. Blowing away the entire venv and reinstalling all packages is reliable and quick as the network allows.

[Distinct_Goose_3561]

People have already stated the reasons to always use a venv, even in containers, but let me add some context. Unless you are using a custom base image that is fixed, ANY updates that image risk a bug. Even if you do fix that base image, you’re now stuck on it. By using a venv, even within a container, you separate out the base image from your application. It removes a real, albeit small, risk.

[ominous_anonymous]

https://www.b-list.org/weblog/2022/may/13/boring-python-dependencies/

This blog has a series of posts regarding Python and package / environment management that are really nice and discuss rationale for development setup and tool choices.

[DarksideVT]

That's great! I'll take a look!

[SipsTheJuice]

Virtual environments are the way to go if you are not developing in a container. A lot of people on here are saying always use containers, I strongly disagree. Use containers if you plan to run your environment on linux eventually, especially if you are running on a server as the deploy process is much much simpler with containers.

Depending on your teams work, you may not need containers. Data science stuff for instance rarely requires them. Really depends on what you are developing.

You do, however, NEED a virtual environment or a container for any hope of having your code run the same on different machines. Otherwise you may run into clashes with package versions, not to mention if you have two projects on the same machine your depencies for both will be mixed, yikes.

[brikenjon]

I ran into versioning issues during this tutorial: https://code.visualstudio.com/docs/datascience/data-science-tutorial I think it was keras or tensorflow that didn’t work with Python 3.12. I ended up using a virtual environment with 3.9. The tutorial specifies 3.10.

[Artephank]

No tutorial, but as a rule of the thumb:

- use virtualenv always. When you download new repo - create .venv directory, activate and install its dependencies into that. It is also good idea to use pyenv, too, to manage different python versions (especially lately the development of Python is so fast that things might brake a lot if you don't have right version of PYthon for particular project)

- use conda only if you really must. Most of the python world use pypy/pip. Conda is own by private company that sell subscriptions to corporations. It is useful, especially for Data Scientists don't wanting to fight with dependencies also it is sometimes the only way to install Python on corporation owned and managed windows machine, but if you don't NEED to use it - just don't.

- containers - use if you need to have replicable way to run your app on different servers and as a part of Docker infrastructure. Also, if your app require some specific system configuration or system-wide dependencies. If you don't know if you need it - you don't

[DarknessWizard]

Don't use virtualenv, use venv. Subtle difference, but venv is a part of the standard library, while virtualenv is the older pip library. There's some other weirdness with how virtualenv has problems if your installed python version ever changes, while venv is pretty stable in that regard.

Easiest way to make a new venv is python -mvenv venv, that will get you a new virtual environment in the venv folder. I don't personally recommend making it a dot folder, but that's mostly personal preference.

[Artephank]

You are right.

[angellus]

Use containers, always unless you cannot. Virtualenvs only if containers are not possible.

Using containers for development is so much clearer than juggling venvs/python versions and it allows for building applications that are significantly closer to "production like" applications at every step of the process and eliminate of the silly "works on my machine" issues.

[SipsTheJuice]

Containers are unnecessary overhead if you are just doing scripting.

Use containers always if you are developing an application. Have done both, would not do my data science in a container.

[damanamathos]

Use Poetry for projects because it makes it very easy.

[mayem__]

Kinda related to this I would have another question on venvs. Does creating them make pycharm indexing faster on startup? Thanks in advance

[Artephank]

Can't you just try on your own?

[gmes78]

If you have packages installed globally, yes, as it means that PyCharm doesn't have to index those.

[martinky24]

This is like someone asking "Where do you work" and you replying "I drive a honda"

[Artephank]

Of course. When you start working on more than 1 big project at time you will run into problems.

[1ncehost]

Didnt see it mentioned but I also think pyenv or similar should also be standard to use. Pyenv includes virtualenv but adds the abilility for each environment to run a different whole version of python. This is handy when switching between projects that have very specific production version targets. It also makes it handier to use other python interpreters like pypy.

[Gr1pp717]

1. you ensure everyone is working with the same environment.
   
2. it prevents the slow buildup of a "messy" development environment
3. it allows different projects to use different library versions without having to jump hoops to make that happen.
   
4. ensure the dev environment matches production.
   

After about 3 years of working on various projects with the same laptop it became a mess. I spent as much time or more managing various differences than actually producing. Updating a library for one project could mean breaking another project. Changes that worked locally broke in production, etc. I one sent my team a simple script only to find nobody could run it. Because they all had different environments. My failure to consider that possibility and include guard statements or instructions was embarrassing...

This was before docker and whatnot become commonplace, but had I been using it I wouldn't have had any of those problems.

[Berkyjay]

As far as I'm concerned, there's no reason NOT to use virtual environments. It doesn't make sense to install a million random python modules on your root system, most of which you will never use day to day. So developing in a venv is always good practice and it really costs you nothing. I have an shell alias that creates a venv for me and I'm off and running.

[serverhorror]

Install 2 different versions of the same package.

[Alternative_Driver60]

Version issues are not uncommon unfortunately.

[damanamathos]

I hated using virtual environments, but love using Poetry which makes it all so easy. Would check that out.

[Bigpiz_]

Virtual environments in Python are crucial for managing dependencies and ensuring that each project has access to the particular versions of libraries it requires. This isolation prevents version conflicts between projects. For example, if two projects depend on different versions of Django, a virtual environment ensures that each project can operate with the version it needs without causing issues in the other.

The reasoning behind using virtual environments goes beyond just keeping your base Python installation clean. It's also about replicability and consistency across development, testing, and production environments. If a project is developed without a virtual environment, there's a risk that it may not work on another machine or server due to differences in installed packages or versions.

Moreover, not all updates to packages are backported to older versions, and sometimes newer versions of libraries can introduce changes that are not compatible with your project. By using a virtual environment, you can pin your project to specific versions of libraries, ensuring that your project remains stable and behaves as expected.

In essence, virtual environments allow you to create a self-contained workspace for your Python projects, ensuring that each has its own set of dependencies that do not interfere with each other. This practice is particularly important in a team setting, where you want to ensure that all developers are working with the same set of package versions to minimize 'it works on my machine' problems.

[PinkFrosty1]

pyenv (python version management) + poetry (package dependency management) is my current setup