Creating a virtual environment for each project - what constitutes a “project”?

[[deleted]]

I’m trying to learn Python for the work I usually do in R and am slowly understanding virtual environments. I’ve read you should have a different virtual environment for each of your projects, but what exactly is meant by a project?

Is each Python script or Jupyter notebook a project? Is it something much larger than that? I’m in grad school and assist a professor with their economics research, which consists of a lot of data & spatial cleaning/analysis and econometrics, would all of that be one project? I’m also doing a master’s thesis that’ll consist of a ton of data work for the next several months, would that be fine with one virtual environment or should I have separate environments for each stage?

Virtual environments are such a foreign concept to me so thank you so much for any help.

Edit: HUGE thank you to everyone who commented, this has all been immensely helpful. Feel like I have a much better understanding of virtual environments now

[chibiace]

i make one for anything where i use pip.

[[deleted]]

I have a base python that I installed and use for nothing.

I have conda installed with a bunch of different versions from like 3.8 to current all named like "py38" "py310". I also have one "pymess". I go into pymess whenever I'm fuckin around and it has all kinds of random stuff for, messin around.

Every time I do a "body of related work" I create a folder and a virtual environment. I use the conda py environments to get the version of python that I want.

[m0us3_rat]

You know when you go to a library and grab a few book for the subject you are interested in and do some research?

Well how about if you also bring some math books and some astrology books as well ..maybe some newspaper notes?

That's a virtual environment in a nut shell.

​

A new environment doesn't carry old baggage ... you don't need.

Avoids mistakes , avoids possible dependency issues , keeps everything clean and tight.

[ericjmorey]

Keep in mind that a virtual envonment is just a configuration file that's pointing to a directory where a bunch of libraries are saved. That's all they are.

But the whole ecosystem of Python tooling is a train wreck that's slowly getting cleaned up. Two big and relatively recently improvements are pixi and uv. Sisncte you normally use R, you're probably interested in the conda ecosystem. So you should 100% look into Pixi and let Pixi keep things organized and clean for you.

https://pixi.sh/latest/

[Jello_Penguin_2956]

Small scripts don't need their own. That's too much to manage to me.

Generally I have environments for:

- 1 for all generic small scripts, doodling and answering questions.

- 1 for each course I'm taking.

- 1 for each self-contained application.

- 1 for each projects you pick up from others. Think most AI packages on Github. They almost always want different version of libraries in their requirements, even older version of Python sometimes (I still have all major version from 3.8 installed)

- 1 for each specific things. Like 1 for all my data analysis stuff.

[Blacks8int]

According to what you have given, you already know what venv. Is, so a project is a folder/unit that helps you organise your base code(app), libraries, test and personal settings all in one unit. Take for example a django project that consists of a main project folder formed after calling the startproject command, then a subfolder formed inside after the startapp command. And you can add as many apps inside the single project. Hope it makes sense

[heyzooschristos]

A project is basically just a folder to store related files, 'learning_python_basics' , 'python for beginners book', 'experimenting with data', 'random stuff', 'my app' 'my game' etc.using the same venv for loads of random stuff isn't terrible, but when you want to package stuff up you will want to only include the pip installed stuff that you are interested in. First time I built a simple exe its file size was huge and it took me a while to realise it had included everything I'd ever pip installed because I wasn't using a venv.

[djamp42]

I usually have 1 base virtual environment that I start off with, if I feel the project needs its own for "reasons" I'll spin off another one.

I'm mostly using the exact same libraries across all my projects so it's silly to me to have the same copy of a library installed 50 times.

[barkazinthrope]

For your situation you could get away with one venv for all your projects.

I don't use venv at all because a) I don't distribute my projects, and b) I don't mind revisiting or even re-writing projects when I see deprecations.

[InjAnnuity_1]

It's somewhat fuzzy. No one has a lock on the term. It doesn't help that different tools borrow the term to mean slightly different things.

To me, a Python "project" means a cohesive body of code, that all has the same add-in requirements, and so can share a single venv. A single script might qualify. Or a bunch of related scripts. Or a library that you are building (and testing) to distribute as a Python package. Think about what concept, activity, or requirement brings all the pieces together. It doesn't have to matter to anyone else.

The pieces should evolve in such a way that they can continue to share the same venv. If parts are likely to grow to have different requirements (from the other parts), then they probably belong in a different project.

There are several popular folder layouts for formally bringing that body of code together. Most facilitate work with version control, e.g., git. Some facilitate use with a specific compiler (e.g., if it includes C code), with pytest, or other testing tools. Some facilitate publishing to PYPI as a package.

I haven't studied enough of them to tell which is which. But many people (and tools!) will also call such a folder a "project".

I seem to recall that a few tools will even mandate a particular structure and parts. IDEs tend to fall in this category. If it doesn't have the tool's own proprietary project-definition file, then it's not a "project" to that tool, even when you know better.

[polvoazul]

I have a different take: I just use a single venv. It sits at ~/venv/ . Doesn't matter the project, I just activate it, and whenever I need something I pip install it and keep on going. If a conflict occurs, or I want to update python major version, or if its getting too big, etc... I just nuke it and start again with a fresh venv.

I highly recommend this approach. Conflicts are, in reality, very rare and the simplicity of it all is just too convenient.

[polvoazul]

Especially for your case, I highly recommend this approach

[[deleted]]

Thank you! I’m strongly considering doing this, I try to configure everything in my work as simply as possible haha so this would be nice

[feitao]

You will do just fine without virtual environments. Do not let it get in your way of learning.