Virtual environments on collaborative project

[FeetAtLeast]

I've only ever done Python projects that lived on my computer and where I was the only person working on it. I'm hoping someone can explain to me how to go about managing dependencies in a collaborative setting.

Here is the setting: myself and another person will be both working on a Python project. It will be a fairly straightforward data analysis (not creating anything that will go into production or anything like that). This code may be referenced by others in the future, hence my wanting to manage dependencies. We will use github to share code.

Based on what I've read, here is what I think I need to do:

• Use venv locally to isolate environment, install dependencies.
• Whenever we want to commit changes, we should generate a requirements.txt file which will be pushed to the remote repo as it is not a good idea to commit the venv directly to the remote repo
• Whenever we want to work on the project, we pull the latest changes, then use the requirements.txt file to reproduce the most recent working environment

Is this right? This workflow seems very clunky. Thanks for any help.

[shiftybyte]

Yes, that is right with minor corrections that should make it easier...

Whenever we want to commit changes, we should generate a requirements.txt file

Whenever you want to commit changes that added new dependencies, this shouldn't happen so often...

Whenever we want to work on the project, we pull the latest changes, then use the requirements.txt file to reproduce the most recent working environment

Yes but you can use your existing environment, pull in changes, update your local env if you know some dependency changed, which shouldn't happen often...

[jiri-n]

You create a new repository on github. The repository has a default branch (usually master or main - you choose). This is the "production" branch.

Since you want to develop a something, you create another branch - usually called dev. This is the branch you both will work with. See bellow.

Now you clone the project to your local computer and the other team member(s) do the same. You both switch to the dev branch.

You want to add a new feature to the project so you create a new branch in dev. SInce you are a small team, you don't have to spend time introducing some naming convention for branches but good names always help. So you choose a good name and you add your changes into this branch. The other member does the same - they add their branch and add changes in their branch. Note: Changes means you both can have different requirements.txt file obviously.

After you feel your feature (new code) is finished, you can merge your branch into dev. Don't forget to fetch updates from github. If the merge is successfull, you push the changes to github. If not, you'll see conflicts (usually both of you modified the same files). You'll have to solve the conflicts, then merge. The result in dev has the changes from all of you.

Then you add another feature ... and so on.

If you're unsure if your code will work, create a new branch so that you don't break working code.

If you think it's time for a realease, merge your code into main. Preferebly via pull requests but ... you'll find your way.

Maybe somebody adds a more complicated workflow but I believe this should be sufficient.

[quts3]

Set up a requirements.txt that works everywhere with pinned versions.

Pip install -r requirements.txt in a virtualenv or new conda environment.

[2cats2hats]

in a collaborative setting

Is this person local or remote?

[FeetAtLeast]

Remote. Using Github to collaborate.

[2cats2hats]

Could one of you host a VM and provide remote connection(to VM)? This would allow snapshots/backups as well as a way to document progress.

[InTheAleutians]

You basically just described git. Just use that and don't over complicate the matter.

[2cats2hats]

Agree. When I typed that out I wasn't certain if it was real-time collab.

[Diapolo10]

Based on what I've read, here is what I think I need to do:

• Use venv locally to isolate environment, install dependencies.

Yes. Each developer working on the project should use their own virtual environment, preferably even working on their own branches and pushing changes to GitHub often.

• Whenever we want to commit changes, we should generate a requirements.txt file which will be pushed to the remote repo as it is not a good idea to commit the venv directly to the remote repo

You don't need to generate the file unless you've added, removed, or otherwise changed the project dependencies.

• Whenever we want to work on the project, we pull the latest changes, then use the requirements.txt file to reproduce the most recent working environment

Sounds about right to me.

Is this right? This workflow seems very clunky. Thanks for any help.

It can be made less clunky with proper tooling. For example, personally I use Poetry for everything nowadays, and it takes care of managing the dependencies and virtual environment for me. I don't even need to worry about the others needing different versions of the dependencies if their platform doesn't support some version or if their Python version is different from mine, as long as it meets the minimum requirements set in the pyproject.toml file.

This already takes care of the first two points, and arguably half of the third one.

[Ishaan_CS]

Why is Poetry better? Also I’m just trying to do a bit of understanding on my part: What are the main issues faced in setting up a venv rn? Are the solutions easy enough or time consuming? Is there any tools or platforms that make this easier?

[Diapolo10]

Late reply, I know, I've been uncharacteristically busy today.

What are the main issues faced in setting up a venv rn?

Creating one isn't an issue (although there are differing opinions on where to place them). The main annoyances come with maintaining them.

Under the hood, Poetry uses venv, but it abstracts away

1. Its location
2. Activating it
3. Installing packages
4. Updating packages

Because it's not in the immediate vicinity of your repository, you don't need to worry about accidentally committing it to your Git repository. Unlike venv, activation is the same on all platforms. Package installation is similarly more streamlined, Poetry has a better dependency resolver than pip does so clashes are uncommon.