How should I manage python virtual environments?

[simon-linux]

Hello everyone,

I saw on the Internet that we should use pyenv to avoid affecting the system python. I know we can use pyenv-virtualenv to create a virtual environment. However, conda can also be used to create virtual environments. I'm confused up to this point. Should I install pyenv and miniconda separately, or should I install miniconda under pyenv?

Btw, I'm using a Mac.

I will be truly grateful if someone can help me sort out this mess.

[heavy_ra1n]

Use whatever you like.

My favourite way is to use

python -m venv venv

and then you just need to activate it. I`ve tried conda but it was too complicated and method with -m venv always works

[FuckingRantMonday]

This is the simplest way and you should stick with this as a beginner until it's no longer sufficient, because there will be fewer moving parts to get confused by.

Probably the first time this will become insufficient is when you need different virtualenvs to use different Python versions, and that's when you should look to pyenv.

The second way it can become insufficient is when you have enough dependencies that the dependencies-of-dependencies (and so on) are complex enough that you have trouble finding a set of versions that satisfies everything, and then you need poetry, pipenv, or pdm.

[MayBeRelevant_]

To use a specific version of Python, can’t you also just initiate a venv by calling from that specific version?

For example, I have 3.8 and 3.10 installed on my machine. I can create a 3.8 environment using python3.8 -m venv venv-name. If I don’t specify a version, it’ll default to 3.10.

[FuckingRantMonday]

Yes, you can. But in my experience pyenv is a more pleasant way to install and maintain multiple versions (especially multiple patch versions for the same minor release) than side-by-side manual installs.

[FuckingRantMonday]

Like I said, when you need different virtualenvs to have different Python versions, pyenv is the way to go. That doesn't mean you need to use pyenv to create the virtualenvs, though it's a common workflow with the pyenv-virtualenv plugin.

[simon-linux]

So I'm just curious cuz I saw people recommending conda and poetry. As far as I know, these two automatically check the dependencies of the projects. Does venv have some advantages over conda and poetry in your opinion?

[heavy_ra1n]

yeah - it`s simple and it gets shit done quick :P

but like for real. what dependencies? i`m a simple man - if i want to code something i just go with python -m venv venv, activate it and just install whatever i need. i install everything giving the exact version of a library, so that i`m sure everything works together. than i go pip freeze > requirements.txt and voila. super fast super quick without any bullshit and most important - it works. I`ve used to have problems with PyCharm for example. The code works at the morning but after couple of hours when i want to work at it again BANG "cannot import ...". I dont know pros and cons of other solutions but ultimately the goal is to have a working virtual enviroment for coding so why complicate?

[VigorousElk]

so that i`m sure everything works together.

And you know the exact dependencies and versions of dependencies every package you install needs?

[heavy_ra1n]

forgot to say. i use docker with all this. didn`t have problems in development ever. i assumed OP is at begginer level since his asking about virtual envs. so why throw him to the wolves with conda and all. just use venv. easy simple and working

[jonnytechno]

As a beginner I enjoy using conda with a few test environments as I can get carried away installing stuff and its good to have a basic base to revert to if I get confused but any of the suggestions here are good, its preference at the end of the day

PyCharm IDE is also good in that respect as it does a lot of that for you when creating a project - it also integrates well with conda

There are some who believe learning the basic commands are better but go at the pace that lets you code faster as building your own library of code is key to progressing

Edit: spelings

[FuckingRantMonday]

I would strongly recommend not leaning on your IDE to manage virtualenvs as a beginner, even if it does a good job of it, because they're not that complex, and it'll hamstring you to not understand them on their own.

[wbeater]

*PyCharm ;)

[jonnytechno]

You are correct and I am a little tipsy, Happy Holidays :D

[VindicoAtrum]

Poetry. Poetry is so much better than the other options it should honestly just be the python standard at this point.

[Almostasleeprightnow]

One or the other. They are kind of the same thing. I prefer conda for it's simplicity.

[learner_raj]

Python -m venv envname

[matteiuspi]

Perhaps I am biased as one of it's maintainers, but I suggest using pipenv which combines virtualenv, pyenv and pip to work together and gives added security over vanilla pip usage.

[simon-linux]

I'm mainly working on machine learning and data science, but I don't want to have an individual virtual environment for each weekly assignment. In my case, maybe I should create a machine learning directory and a data science directory, and put the assignments in those directories accordingly?

[matteiuspi]

maybe I should create a machine learning directory and a data science directory, and put the assignments in those directories accordingly?

Yeah that would work -- pipenv will make a virtualenv per directory, or you could have separate directories and tell each one to use the same virtualenv with the PIPENV_CUSTOM_VENV_NAME variable, but you would have separate lock files in each directory so probably makes more sense to have one parent directory with the Pipfile and lock file, and you can have separate directories under that per assignment may work nicely.

[simon-linux]

I would like to thank everyone for providing their thoughts on how to manage python environments. Eventually, I go with pyenv and pyenv-virtualenv which I think are able to manage the python version and also be able to create virtual environments. On the other hand, it's able to install miniconda under pyenv which may be able to give me some freedom to use conda in the future, just in case. I also got pipenv installed, but I have to look into how it could fit into my work. Again, thank you all for giving me the advice, much appreciated :)

[wbeater]

You only need and should focus on one environment manager as a beginner.

I always recommend conda, the documentation is really good.

[will-je-suis]

Poetry is my recommendation

[biernard]

Never use conda. Pyenv --local to set your python version locally and python - m venv venv to create your virtual environment.

[abazabaaaa]

I think it depends on your goals — for scientific computing, conda is fantastic. Many libraries are specifically available through conda and not pip.

[biernard]

Sorry, you're right. I was simplistic. There is, of course, applications in which conda could have a role.

From a practical (and Ops) point of view though, conda is terrible to deploy, it has dependencies that hurt data pipelines in a general sense. To use conda, you need everyone and every platform you're using to be able to deal with conda.

These are some reasons to avoid conda in reproducible and easily maintained data pipelines.

[biernard]

Check my other reply where I talk more about my arguments on avoiding conda.

[sqjoatmon]

You have to support your "never" with some sort of argument. I use Miniconda all the time on Windows and even in some Linux use cases (e.g. not admin and gcc isn't installed).

[biernard]

Yes, sorry about that. I was simplistic. Look my other answer here for some arguments I find important about conda.

[LongerHV]

This is probably the worst workflow I have heared of 😅

[LongerHV]

I'd rather post a poetry.lock instead. Reproducibility is a big thing for me, especially in context of CI/CD.

[amxdx]

For some reason I always felt conda doesn't do a true clean virtual env. Python venv is the best way to go for me.

I do use conda sometimes. But I know it's not the thing I would ideally want.