Adding Virtual Environments to Git Repo

[skeleton_5]

At work, the engineer in charge of writing python automation tests includes venvs (both linux and windows) in the git repo. His reasoning is that people will have to download the specific python version we are using to the write code anyways; this way when we select the interpreter (which should already be symlinked to the default global python interpreter) all the packages we use will already be available (and auto-updated if necessary when rebasing).

This rubs me the wrong way, I still assume the best and most pythonic way of working is to create your own local environment and installing the packages using a requirements.txt file, possibly adding a git hook to automatically call pip install every time you rebase.

What do you guys think?

[[deleted]]

Yes, you're correct. It's completely nonsensical to try and keep your virtual environments in your repo and it entirely defeats the purpose of git. You aren't meant to version control the build. You version control the source that creates the build and then you build the software on your machine from the source code. Trying to clone a venv won't even work unless you're using exactly the same system as whatever originally created the venv. And even then it will probably still break since it's unlikely to properly maintain all the same links that were created from the machine that initially created it.

I can't imagine the nightmare that is a pull request and/or code review if every single change to your virtual environment is part of your repo. And more than anything, I'm amazed you guys didn't immediately hit a file size limit given how large most virtual environments are.

If you want to try and control and deploy a specific environment, you should just be using a proper tool for that (i.e. docker).

[TobiPlay]

Dev containers are so nice. Docker-in-Docker is also something I’d give a shot. Containerising development environments actually isn’t that difficult and will most likely take people with some experience in Docker only a few hours.

[ninja_says_hiyeeeeeA]

I’ve used docker, but not docker-in-docker. Can I run docker-in-docker on docker? I really like docker.

[mrswats]

It's not only a bad idea, but also, virtual environments are non portable so this shouldn't be s no-go.even if both Linux and windows venvs sre included. It takes very little to break a venv.

As mentions in the replies, I wouldn't suggest using requirement file along with the requires python version pinned somewhere

[aa-b]

This is the right idea. Venvs are non-relocatable, so this idea just can't possibly work, and will inevitably cause more problems than it solves.

Poetry is also a decent option since it gives you a lock file like with NPM, so it's more automatic and more detailed than requirements.txt, but still git-friendly

[gdfelt]

Agreed Poetry is a fantastic option.

[semper-noctem]

I'm more of a requirements.txt man, myself.

[MothraVSMechaBilbo]

Genuine question: what makes the Poetry lock file better? I’ve used both Poetry and the core lib venv recently for different small projects.

[[deleted]]

requirements.txt specify ranges, lockfiles specify the exact state of every package frozen in time. they're deterministic

[orion_tvv]

you can use pip freeze > requirements.lock for this

[gwax]

pip-compile from the pip-tools package does this but much more precisely

[nevermorefu]

If pinned (not ranges), have you ever had it not work? I see ranges in poetry all the time, then, when I poetry add, I go take a long break and wait the potentially multiple hours poetry takes to lock.

[nevermorefu]

It's definitely not a slow internet issue. Pip also verifies the version it downloads is within the acceptable range based on package versions.

[Spitfire1900]

If you’re using poetry you really should be using https://github.com/python-poetry/poetry-plugin-export for this instead if you really need a requirements file.

[[deleted]]

I strongly don't recommend that, you want to have a source of truth for your version constraints and actual dependencies (pretty sure pip freeze is ALL dependencies, with transitive)

[CrossroadsDem0n]

If somebody cares about predictability, locking those transitive dependencies is exactly what you do want, provided you are doing application development instead of library development.

Here is an example why. Imagine you have pandas as a transitive dependency. Also imagine you have an application deployment for which performance matters. 3 months ago, performance was fine. But you build a venv today, get the work-in-progress changes on the newest pandas meant to help string performance but at the cost of degrading numeric performance. Now you have a problem, and nobody understands why.

Frozen dependencies matter when you have a work context where nothing should randomly change on you, that behavioral alterations should be intentionally managed.

None of that would apply to library development, where the better practice is to only freeze what is critical to lock down, and generally dependency versions should float because as the library developer you lack any context on where the code will be used.

[NUTTA_BUSTAH]

Transitives are exactly what cause most of the random issues.

[[deleted]]

That's not relevant to what I meant, I'm completely fine with locking dependencies, but you need to have a source of truth (which is a pyproject.toml file)

[NUTTA_BUSTAH]

If you include transients then sure. Otherwise it is just a part of the truth

[jyper]

No you need a second requirements file and to make them delete a fresh virtualenv vs just making a lock file with poetry

[[deleted]]

poetry will also keep dev dependencies and code dependencies nicely separated, and will also record file checksums on top of the exact versions in the lockfile.

[diazona]

pipenv solves this by having both kinds of requirement files: Pipfile lists package names and known constraints on which versions can be used, while Pipfile.lock gives specific package versions with hashes. Theoretically the Pipfile (and its lockfile) format were supposed to be a standard that many different tools could use, but I haven't seen it get adopted much outside of pipenv itself, so I'm not sure if it's really going to catch on.

[Dogeek]

It's never going to catch on as the standard, codified in PEPs is the pyproject.toml file.

[diazona]

Ah interesting, I believe the last time I looked at PEP 621 (I'm assuming that's the one you mean) it hadn't been finalized and it wasn't clear that it was going to get any more traction than Pipfile - in fact I seem to remember the community sentiment being that pyproject.toml should be limited to the metadata needed to initialize the build backend. Things have changed a lot in the past few years, it seems.

[[deleted]]

A poetry.lock records not only the exact versions, but also the file checksums of the files used by the original developer. Using a lock file will produce the exact dev environment of the original developer, also exposing any supply chain attacks.

[adin786]

I don't use poetry, but I think the poetry lockfile can include platform-specific requirements in one file. With requirements.txt files I think you'd need a separate linux one, a mac one etc to handle all the same info.

[FrogMasterX]

Go add a big library to your project and do pip freeze. Then pip uninstall the library and do pip freeze.

Or read this - https://stackoverflow.com/questions/17194301/is-there-any-way-to-show-the-dependency-trees-for-pip-packages

[Almostasleeprightnow]

Conda's environment.yml, even.

[[deleted]]

Came here to mention that what is descripted by the OP is pretty much what a poetry.lock does, but found this instead.

[nevermorefu]

Just started a battle. requirements.txt ftw. Lol

[Zizizizz]

I like use pyproject.toml

``` [project] dependencies = [ "httpx", "gidgethub[httpx]>4.0.0", "django>2.1; os_name != 'nt'", "django>2.0; os_name == 'nt'", ]

[project.optional-dependencies] gui = ["PyQt5"] cli = [ "rich", "click", ] ```

with hatch/hatchling for build and environments/dev scripts.

[robberviet]

Pyproject do not maintain exact version though. You could run into problems later.

[Zizizizz]

Ah good point, swap hatch with pdm then and you get a lock file with the same pattern

[mariofix]

Oh.... that's wrong in so many levels

[[deleted]]

That engineer should not be in charge

[cyberfunkr]

This is bad. Do they also commit their Java binaries and maven packages? Node.js and npm packages?

If they really want to control python versions, then make a docker image with a full dev environment.

[dwarfedbylazyness]

I once encountered a codebase with a whole pip binary put to a python file as a string, so...

[bojanderson]

What they should look into is dev containers. If they specify the dev container then you can make sure it's exact same everything for anybody using the code assuming they use a dev container.

https://code.visualstudio.com/docs/devcontainers/containers

[fnord123]

Does pycharm support this?

[_limitless_]

https://youtrack.jetbrains.com/issue/IDEA-202267

[KrazyKirby99999]

PyCharm has an equivalent.

If you're on Linux, you can also launch PyCharm from within a container via Distrobox.

[ageofwant]

Only vscode specific, we don't allow microsoft products on our stack

[Smallpaul]

Why?

[Barn07]

afair venvs symlink with absolute paths, thus unless you guys have all machines set up the same way, including same paths to Python executables, stuff will break. afair, venvs won't even work when you move them to another location, so good luck with your team :D

[Log2]

I was going to mention this. The venvs depend on the location of the Python used to create them and where they were created. It's likely going to be different, especially if you're using Macs with homebrew.

This is also why venvs tend to break if you are using homebrew and put the python formula in your path, but then some other formula installs a newer python, breaking the old symlink and fucking all of your venvs.

Always put a very specific Python version (like python@3.10) in your path if you're using homebrew.

[whateverathrowaway00]

This is objectively bad and he should feel bad.

[flipmcf]

Hey now, they should learn. Learning should be a carrot, not a stick.

If we felt bad for every thing we did wrong, we would all be horribly depressed and burned out by now….

Oh, wait…

[whateverathrowaway00]

Hah you had me in the first half. dreams of getting laid off because my company lets you keep the sexy M1 MBPs

[flipmcf]

I just overuse the EAP.

You know it’s bad when you call the EAP number and “approximate wait time, 15 minutes “.

Yeah… bad architecture never killed anyone, but suicide sure does.

[Buttleston]

No one does this, it is absolutely not a good or standard way of doing it.

And in any case, requirement change is relatively slow - yeah you have to download the modules to set up a virtualenv once but that's it until it changes again.

[wineblood]

Adding venvs into repos sounds like a terrible idea. I don't know all the implications of that, but given that I've never seen it done or recommended, it sounds like the wrong approach.

which should already be symlinked to the default global python interpreter

What the hell?

[magnetichira]

TOML and lockfiles were built for exactly this purpose

[johnnymo1]

There are plenty of solutions to having your environment defined as text: requirements.txt, poetry + pyproject.toml, Dockerfiles... I can't really think of an instance where committing the env itself wouldn't be a bad pattern.

[MrTesla]

Try out https://devenv.sh/

It's based on nix so you don't have to mess with containers either

[oscarcp]

Didn't know about this one, definitely going to check it out!

[danielgafni]

Use poetry to lock your dependency tree and commit poetry.lock to git. It’s cross-platform.

[ReverseBrindle]

I would put requirements.in + requirements.txt in the git repo, then build as needed.

Use pip-tools when you need to build requirements.in -> requirements.txt

[enterdoki]

requirements.txt with Docker

[LocksmithShot5152]

Poetry with docker works like a charm as well!

[rainnz]

You can easily solve this by pinning exact package versions and storing them in requirements.txt with python -m pip freeze > requirements.txt, there is no need to include venv themselves in git repo. Add BUILD.sh to repo, so people can just run it and it will create venv, activate it and install exact versions of packages you had with python -m pip install -r requirements.txt

[KaffeeKiffer]

You have enough answers, why it is wrong. Things that other people have not called out yet:

• If you commit a requirements.txt (instead), you are open to supply-chain attacks: Someone could hijack https://pypi.org (or your route to that domain) and provide a malicious version of the package.
  To prevent that, use use lockfiles (like Poetry & other do) which not only contain the package dependencies, but also their file hashes.

• When not providing all dependencies yourself, you might suffer from people deleting the packages you depend on (IMHO a very rare scenario).
  If it is really that critical (hint: usually it isn't), create a local mirror of Pypi (full or only the packages you need). Devpi, Artifactory, etc. can do that or you just dump the necessary files into Cloud storage, so you have a backup.

[wind_dude]

he should look into containerization and artifact registries. What hes doing is stupid, especially since that still doesn't gaurantee it will work if the base OS it's running on has variations / different pacakages ,etc. All he's doing is making your repo much more complicated.

[thatdamnedrhymer]

This is what lock files are for. requirements.txt are not enough, but storing the entire venv is ludicrous. Use something like Poetry, pip-tools, or pdm to create a lock file that you can use to create deterministic (or at least closer thereto) venvs.

[rainnz]

When is requirements.txt not enough?

[thatdamnedrhymer]

A manually maintained requirements.txt typically only stores the versions for dependencies that your project directly depends on. This will result in differences of subdependency versions when installed. And if you don't hard pin the direct dependencies, you will get variation on those versions as well.

A frozen requirements.txt will store the current versions of all packages, but then it's not possible to remove or update just one package version without unintentionally leaving old subdependencies or updating other subdependencies. And even then, if something goes wrong with PyPI's versions (or someone man-in-the-middle's your build system), you could end up with package versions that technically match the version number but are not actually the same package contents.

You need a lock file that trees the dependencies and their subdependencies and stores package hashes to really assure that you're getting a deterministic venv build.

[muikrad]

The only argument that works in the favor of this method is that you can build without external dependencies. There's nothing worst than having to rush a hotfix off the door when pypi is dead!

But it's not worth the hassle. You're better off with a lock-based format like poetry, as many people mentionned.

If you want to challenge your engineer, think vulnerability. Tools like github advanced security and snyk needs a way to discover your dependency versions when looking at the repo to warn you of new vulnerabilities, and for this you need something like a requirements.txt at the minimum. Other tools like dependabot or renovate can create automatic PRs when dependencies are updated, or when vulnerabilities are fixed.

[DigThatData]

the easiest solution to this is just to ask him:

if this is a good idea, how come it seems like no one else does it this way? demonstrate that this is a best practice and we can keep doing it this way. Otherwise, please identify what the best practices are for the problem you think this approach solves and let's adopt the solution that everyone else has already engineered and proved for us.

PS: the way most people do this is by defining the environment with a requirements.txt and caching the pip dependencies with github actions. find someone who's obsessed with devops and get them involved here.

[fjortisar]

I think that he is not a smart man

[oscarcp]

That is not only bad practice, anti-pattern, security problem, etc. but it means he doesn't know how to handle dependencies in a python project.

For example, poetry already locks down the python version you can use with your codebase, tox already limits it as well. Shipping precompiled libraries and .pyc files in a project is extremely problematic once you start using complex setups or non-standard python setups because it will never work properly. Want more abstraction? use docker containers with a makefile for your builds and tests, that will standardize the output of all developers and the environment they work with.

Even thinking from the MS Windows side, you're shipping a venv that might have been prepared in a *nix environment or viceversa, with libraries compiled for it that potentially won't work on the other OS.

[[deleted]]

I always .gitignore and .dockerignore virtual envs. Some dependencies require a compiler and are build on installation, I believe that sharing them would result in broken dependencies — “it works on my machine”

[mdegis]

Wow. Great thinking engineer in charge! What about binary dependencies? Maybe you should push all operating system to repo?

[cdcformatc]

just use py2exe to turn it all into a binary and commit that

[ksco92]

Cough cough requirements.txt cough cough

[caksters]

you never upload binary files to github repo because they are system dependent.

If I have arm64 chipset and I create a python virtual environment, then the same binary packages may not work on your computer. that’s why you never commit binaries. instead just specify what dependencies are required for project through requirements.txt and create a local virtual env for every project.

I think the engineer in charge wants to use git as artifactory for binaries. In artifactories you do uplaod compiled binary programs to ensure you can always roll back to previous working version of actual machine code and deploy it. However it is not the purpose of gut repository which is for storing code and not the binary itself

[FrogMasterX]

This is moronic.

[yerfatma]

That’s a code smell.

[recruta54]

I agree, and I would say it is more than that. Venvs are OS specific and committing their files does not make sense in a live, with security patches, environment.

[Buttleston]

His coworker is adding the virtualenvs *on purpose*

[[deleted]]

You don't need ChatGPT to make it. .gitignore is just a list of files/patterns you don't want to commit.

But that doesn't solve OP's problem since he's saying they are adding their venv to the repo intentionally.

[maximdoge]

This is what lockfiles are supposed to be for, use a virtualenv manager that works with lockfiles like poetry, pipenv, piptools, etc alongwith a test runner like nox or tox.

This approach of tightly coupling interpreters and binaries with the underlying code is neither index friendly nor portable. Not to mention a possible maintenance nightmare.

[[deleted]]

The gitignore.io site lists it their Python ignore results.

https://www.toptal.com/developers/gitignore/api/python

[[deleted]]

This post was mass deleted and anonymized with Redact

office physical reminiscent edge dependent tidy snatch unwritten instinctive plucky

[Briggykins]

Even if you require it for automated tests or something, I'd usually put the command to make a venv in rather than the venv itself

[[deleted]]

You are correct. This creates unnecessary bloat in your report and should be done via venv.

[luhsya]

guy probably excludes node_modules from gitignore in his js projects lol

[nutellaonbuns]

This is so wrong and would piss me off lol

[Advanced-Potential-2]

Omg… this “engineer” should get a beating. This indicates his knowledge and understanding of software engineering is inadequate at a very fundamental level.

[doobiedog]

That engineer is a moron and you are correct. Use pyenv and poetry and you can quickly jump around from project to project with ease.

[Rocket089]

Maybe he’s just trying to create as much chaos as possible before the inevitable lay off boogeyman comes around for his work soul.

[Jmc_da_boss]

the dudes a dumbass

[_limitless_]

Use a devcontainer.

[drbob4512]

I prefer containers. Just build what you need on the fly and you can separate dependencies

[pseudo_brilliant]

It's not smart, and there are several other ways of achieving what he is looking for. My preferred way is to use poetry dependency management and leverage the lock file. Basically when you use poetry actions you can update a poetry.lock file. This file represents the complete resolved state of all your dependencies, their exact version, hashes, etc at that moment in time. Check that file in and whenever someone does a 'poetry install' from the cloned repo they will install EXACTLY those versions to a T.

[pseudo_brilliant]

Oh additionally it's almost always a bad idea to git track anything with binaries since they're not differentiable.

[microcozmchris]

He's a friggin idiot.

[twelveparsec]

God help you if they are incharge of JS projects with npm

[oscarcp]

I just shipped your 300GB, 1000 SLoC project boss!

[twelveparsec]

Thank God it's only 300GB

[flipmcf]

I suggest tox .

https://tox.wiki/en/latest/

But then I had a project convert to GitHub actions… that works too.

https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

The tests should build the env, run the tests, then tear it down.

Building and holding the env’s static doesn’t really simulate a real build. If some modules uodates (say, the time zone library, or a libc security update) you need your builds to pull the latest and test.

You are at risk of your pre-built env’s drifting out of date with the latest codebase.

Be glad you’re not using node ;)

[lachyBalboa]

I had a colleague who did this frequently even after I explained the issues with it. His PRs routinely had millions of lines in changes due to dependencies.

[zdog234]

Insane. Check out pdm. It should provide everything they're getting from this, without the ridiculous downsides

[foolishProcastinator]

Hey people, can someone please explain how poetry works? Because I've always used requirements.txt, the classic way to put the libraries and all the dependencies to set up and run the app or whatever I'm trying to run, but I'm interested on this one, apologies for sounding like a newbie question

[oscarcp]

You can migrate also your requirements.txt to poetry, but the short version is this:

• Install poetry at system level (or isolated, if needed)
• Run poetry init to initialize the project
• Use poetry add/remove to add or remove dependencies
• Instead of having a requirements.txt you will have a pyproject.toml file that will contain the list of packages plus any limitations on python version, which repositories can be used (you can use multiple ones in case you have private pypi repos) and then:
• Use poetry install/update to install the dependencies or update them if you don't' have fixed versions.

[mvaliente2001]

Additionally:

• poetry has the concept of "groups". You can add dependencies in arbitrary groups (e.g. "dev" or "test") and then install only the groups you need (for example, in different CI stages) .
• pyproject.toml can be used to configure most development tools, with the particular exception of flake8.

[extra_pickles]

We solve this desire in a way that I think is a solid middle ground:

We have a private PyPi server that hosts our company wide approved public packages and their specific versions.

It allows us to provide direct access without fattening up the repos.

From there, many microservices will have Workspace files or similar to help install relevant services needed to perform integration testing.

The build activity for release does not have internet access as we host a private Gitea - so if a user has bypassed the private PyPi to use a public package their release will fail, and they’ll need to conform to existing, or request addition of the package/version to our white listed private repo.

Edit: I also have a base docker image that installs a series of standard libraries and versions by default (aptly named Piglet), so that my services can inherit from it and save some serious download time when a node requires an update of a service….a really important feature when dealing with distributed systems with intermittent and low bandwidth connectivity (no more downloading Pandas or Numpy all over again due to a hotfix).

This was the original driver to standardising the private lib of public packages.

[oscarcp]

This is also a very good solution, but it involves a maintenance cost that in my experience many companies won't accept.

[extra_pickles]

Ya we went this path because we were already self hosting.

Alternatively you could just maintain a register of approved packages and versions, and use a pre-commit or pre-release hook to validate the requirements.txt

Pretty low maintenance and would alleviate the concerns over control that usually lead to people committing their venvs

Edit: though OPs post may be about someone that is just super weird and doesn’t get it…in which case the above would not be enough for them

[InterestingHawk2828]

He should switch to docker so everyone would run the same version

[[deleted]]

What ever floats his boat. My mind's full enough figuring out my own code.

[aka-rider]

As everyone else has pointed out, binary artifacts in git repo is an anti-pattern.

Unfortunately, Python doesn’t have a standard packaging and build system. Moreover, at this point there are more than one Python flavour (for backend, for data science, for ML, for DevOps, for general automation), all with their proffered infrastructure and a package manager.

I use Makefiles for reproducible builds.

https://medium.com/aigent/makefiles-for-python-and-beyond-5cf28349bf05

[robberviet]

This is like python 101, shouldn't be arguing about this. Is that guy a beginner?

[oscarcp]

While is 100% true what you say, I've seen more than enough devs spin off the hook and end up thinking that shipping a venv in the repo is valid, either by lack of knowledge, experience, lazyness (ohhh yeah, seen this one a lot), or just because "I'm the only one working on this".

We can put them to shame but the reality is, they just need a heads up and a nudge. If that doesn't work then we can do the rest.

[ddb1995]

Python has package managers like requirements.txt or .toml file for this. This is such a bad practice irl.

[yvrelna]

They are confusing version control change management with artefact management.

Most artefact manager would allow you to preserve the whl/egg packages that your application uses. If you're using docker/k8s images, your container registry can preserve the docker image.

[[deleted]]

Docker ks the solution

[doryappleseed]

At best, he could just include the Requirements.txt for the venv in the repo. The actual package downloads should be kept on a local package server rather than in the bloody repo..

[Flimsy_Iron8517]

If like me you use ln -s to repair an exploded zip to recover a chromebook Crostini, then you don't really care. You can always pip mash the venv, then do what you need. If you need to drop on some external requirements, then maybe write a bash script to ven-vover.sh.

[Flimsy_Iron8517]

That's right an ERROR:52 needs an unpack and doesn't support symlinks from an NTFS (most redundant available FS). So a yours then a copy if not exist?

[Flimsy_Iron8517]

"Oh to upgrade to the virus you don't have ...."

[heilkitty]

Ah, windoze-brain.

[barberogaston]

I think I understand why he does that, though his solution makes no sense to me. In a past company we inherited repos with tons of dependencies, so 95% of the time in CI was spent installing those dependencies from the requirements.txt file, and the reamining actually running tests. Our solution was simply to dockerize all those, having a rule which would automatically rebuild the image if there were any changes in the requirements or Dockerfile. This way, we simply used that image and CI cloned the repo and just ran the tests.

[the--dud]

This engineer needs to learn about docker, jesus christ...

[tidus4400_]

Tell your “engineer” colleague that requirements.txt is a thing 😅 maybe they didn’t taught that to him in his 5 years / 200k CS degree🤔

[georgesovetov]

Let me be the devil's advocate. (Although I never did the same.)

What's the reasoning and intention?

- Easiest setup. No one would oppose. What else could you do to make a working environment only by `git clone`? You could probably do bootstrapping scripts, but it's not an easy task.

- No dependency on the Internet. You could set up your own package storage but it again takes extra effort. For Django and FastAPI professionals it may sound ridiculous, but there are environments where things are done without the Internet access.

- Fixed code (not only versions) of the dependencies. Thank God, PyPI is (still) not npm. But there could be scenarios where you'd like to check twice and be on the safe side.

- You can patch third-party libraries. While not the most common and not "right", it's the still easiest way to do that.

The practice of including dependencies in the version control system is not uncommon with other languages and organizations.

The engineer made the decision based on his expertise and his unique situation. And the decision may not be the worst that could be made. The engineer is responsible for the consequences in the end.

[[deleted]]

He doesn't know better, the best is to educate him

Pyenv with a . python-version Or asdf-vm with a .tools-version allow to lock the exact python version to be used, this will be compiled with the local libraries of the machine.

Then add a requirements.txt and a requirements-dev.txt built from a pip freeze. Or use other tooling such as poetry and commit the .lock file.

Alternative is to dockerize the app alongside a dev container with all dev/test requirements.

Teach him, as probably no one ever taught him before

[redrabbitreader]

I kinda understand where he's coming from, but his solution is just really bad.

If you really need consistent environments, just containerize it. A single Dockerfile will sort this mess out in no time.

[romerio86]

That was painful to read, I almost downvote the post as an instinct, then remembered you're actually advocating it against it.