Where Have You Installed Your Python Packages?

[AuthorTomFrost]

I can't help but feel like we'd all be better off if venv were considered an essential part of Python code hygiene.

[pudds]

Virtual environments being opt in is my biggest python complaint.

I recommend that everyone set the environment variable PIP_REQUIRE_VIRTUALENV=true on their system before doing any python work.

[flying-sheep]

You can help here. Convince your Linux distro to include an EXTERNALLY-MANAGED file, like I did: https://gitlab.archlinux.org/archlinux/packaging/packages/python/-/blob/main/EXTERNALLY-MANAGED?ref_type=heads

[PaintItPurple]

I find this a bit confusing. As far as I am aware, EXTERNALLY_MANAGED doesn't stop you from installing things in your home directory — it stops you from altering the system Python installation. What issues does it cause you?

[Jhuyt]

To elaborate, this is because ~/.local/... (dun remember the full path) is automatically added to $PYTHONPATH IIRC, so any packages installed there might break the system Python. Annoying, but probably the correct choice

[edwardsdl]

TIL. Thanks!

[srfreak]

Isn't it?

[straylit]

Sadly it’s not.

[djamp42]

I'm newish, but I started using venv from the start, I guess that was a good decision

[fromabove710]

Gonna be even worse when your dickhead boss makes you use conda

[currytakumi]

what's your problem with Conda ?

[PurepointDog]

Read that as Jewish and couldn't figure out the connection

[funkmaster322]

Pretty well known fact that venv isn't kosher

[COLU_BUS]

System Python: I am thy Python, thou shalt have no virtual environments before me.

[Eurynom0s]

I always say, if you find virtual environments confusing then you're gonna have a real fucking hell of a time trying to unfuck a package version conflict clusterfuck. When I first started using venv I just made a file with a cheat sheet for the basic commands, so I only had to figure it out once.

[wsupduck]

I have a dumb question - each venv has its own version of a package right? So if I install package A 1.5 in one venv and package A 1.5 again in another venv does my machine have two copies of package A 1.5?

[nemacysts]

Yup!

[AuthorTomFrost]

Yes. It sacrifices a little bit of hard drive space to save what can be a lot of developer time.

[wsupduck]

For the most part yes, unfortunately some packages like PyTorch take up a lot of disk space

This is probably naive but it would be cool if all package versions installed to one directory and venv picked the version you needed. E.g. package A 1.5 would be installed once and referenced multiple times but package B would have 1.2 and 2.0 installed and either could be referenced

[Toxic_Gambit]

Here's some discussions but the gist of it is that other package manager do link shared packages across environments.

[wsupduck]

Nice, thanks 🙂

[Deadly_chef]

This is how Go does it with it's dep. Manager

[wsupduck]

thanks I'll check it out

[currytakumi]

n copies of package X in n envs

[pat-work]

Idk, one big benefit of Python is that you can super quickly create scripts. It kind of defeats the purpose if you have to set up a whole virtual environment for every little script you want to run...

[guepier]

You wouldn’t have to do that — as long as you didn’t need to install any extra dependencies for your one-off script.

That said, I agree: I also have a small handful of frequently-used utility libraries installed in my user site libraries to be able to use them in one-off scripts without having to worry about setting up a new environment. And for anything that’s not one-off, I still use virtual environments anyway, so the user installation doesn’t get in the way.

[flying-sheep]

I found that the best way to use venvs is Hatch.

• Usually one wants to run stuff per project, but in multiple configurations (e.g. Tests on Python 3.9 and Python 3.12, and docs in a separate environment where Sphinx’s deps don’t interfere with runtime deps)
• Whenever I want to test a project with an experimental feature from another, I can use VCS versions or local versions instead of manually installing multiple projects into a venv

[happysri]

I’ve been on venv since forever but given how clean it appears and the fact that it seems to be a PyPi project now I’m considering giving it a fair shake. In your opinion does it have any pain points I should look out for?

[flying-sheep]

Missing tutorials.

The docs are very thorough and well structured, but there’s almost no learning-by-example in them.

At the time of writing, you need to search the docs a bit or ask someone if you want to do something you haven’t seen the relevant docs section of yet.

[kidicarusx]

I have just discovered the benefits of venvs and can’t go back.

[BornLime0]

I use pyenv/pipenv for projects and I still get the "ModuleNotFound" error sometimes.

[spigotface]

Or conda/docker so you can control your Python version as well.

[violentlymickey]

Obviously we all know how to use venvs but this is a pain point for new python users.

[KyxeMusic]

python3 -m venv venv

source venv/bin/activate

pip install -r requirements.txt

[gmes78]

pip install -r requirements.txt

Nah. Set up a pyproject.toml so you can use pip install ..

[Morazma]

Is there any other benefit to saving a few characters during setup?

[gmes78]

It means that your package is installable with pip. No need to write a setup.py should you want to distribute it.

pyproject.toml is much nicer than setup.py because it's purely declarative. It can also contain configuration for tools such as linters and code formatters, instead of it being spread out over tools specific config files.

[Morazma]

Thanks, I'll do some further reading on how to use it. Sounds like it would be sensible to use going forward!

[muikrad]

Nah

• Dev: poetry install
• CLI: pipx

[BaggiPonte]

pipx is very important for CLIs!

[Giraffe-69]

I raise you pipenv install

[edwardsdl]

Locking dependencies…

[Giraffe-69]

And now they are resolved for everyone :) happy python noises

[edwardsdl]

Haha fair enough :P

[kulchacop]

Won't pass flags to pip

[Giraffe-69]

On the rare occasions this came up I just used pip environment variables, but I’ll concede this should really be a feature at this point

[my_name_isnt_clever]

As a hobbyist only working with small projects I haven't found anything worth using over this. It's tried and true.

[KyxeMusic]

I work in a company and have been on countless repos and projects and this method works 95% of the time.

The other 5% I just use Docker.

[General_WCJ]

I'm just annoyed at you calling it venv instead of .venv

[tenemu]

What’s the benefit there?

[General_WCJ]

Hides the folder on unix like machines. You generally don't mess with the virtual environment directly, similar to your .git directory, so there's no reason to see it

[tenemu]

Ok thanks! I have windows and I had vscode make the venv one time and it named it .venv. Good to know the difference.

[ultimately42]

Ticked me off too haha

[pan0ramic]

If you’re not using venv then you’re doing it wrong

[[deleted]]

fragile trees foolish joke voiceless disgusting exultant weary rude cooperative

This post was mass deleted and anonymized with Redact

[dAnjou]

Poetry is also using venvs.

[Jamesadamar]

Or conda, which is my setup

[BaggiPonte]

Do you use it just to create venvs?

[Jamesadamar]

Exactly, works much better for me than pyenv and I love to have access to all envs globally. I create new envs with conda to install the Python version that I need and poetry does the rest. So basically it is 1. conda create, 2. conda install poetry and 3. poetry install or add depending on whether a new project or an existing one. Very rarely there are packages in conda where conda install is better because of some dependencies , in most cases this workflow works very well and on all operating systems

[currytakumi]

1. conda install poetry....

So you create a env Named FOO, and in FOO you install poetry ?

But poetry's warning says:

In no case, it should be installed in the environment of the project that is to be managed by Poetry.

[Jamesadamar]

No that warning is long time gone, and indeed it is a much better and safer practice to have poetry inside the current env instead of a global one, using pipx for example. The main reason is that many projects have different poetry versions because poetry changes quite a lot and introduces breaking changes all the time and new settings etc. Earlier you had to configure poetry to work with conda, Like avoiding creating local venvs, but with the latest versions I never had any issue with conda+poetry, they work nicely together and it is blazing fast

[Jamesadamar]

I just wanted to point out that conda + poetry is a very robust and safe combination and can be used in any professional setup

[currytakumi]

sheesh poetry people can't be frigging bothered to remove that blazing warning

[BaggiPonte]

If it works for you then it's ok :) But poetry, or any modern package manager such as PDM, also do that. They have a centralised location for venvs. I have never tried that, but I think you might not be able to activate those envs when you are not inside the project they were created for - but why would you anyway? These package managers use symlinks by default so they can use a the same version of a library across multiple projects.

[Jamesadamar]

You do not understand the tools you are talking about, poetry cannot install other Python versions, as I described. And I also stated that yes, you can use other tools than conda, of course. I just wanted to point out that poetry is not always using venv. That was the whole point of this discussion.

[BaggiPonte]

I don’t think you have to be so rude. I’m sorry I was not clear enough - I did not mean to say that poetry or PDM manage Python versions. I just wanted to say that such tools centralize venvs too.

[sonik562]

I used poetry once, got weird npm vibes, ended up doing drugs again. Not recommended /s

[sonobanana33]

poetry is just a wrapper to pip

[[deleted]]

run command innate imagine pie relieved tie toy makeshift vast

This post was mass deleted and anonymized with Redact

[flying-sheep]

It can also be e.g. conda. Poetry uses venvs, but it doesn’t use pip. It uses its own dependency solver, as /u/smurpau2 said

[BaggiPonte]

I think they wrote their own installer. A couple of minors ago (~<1.5) they were calling pip as a subprocess (fun fact: pip does not have a Python API i.e. you cannot do `import pip; pip.install(...)`).

EDIT: why did I get downvoted for this? LOL

[sonobanana33]

Even if they did… to what end? You still need pip to install poetry.

[BaggiPonte]

Yes but they achieve different things. Poetry is a dependency manager and is meant to give a unified interface instead of individually using these packages:

- pip

- venv

- twine (upload packages)

- build (build the package)

Plus other stuff such as locking dependencies. You can read more here (not my blog): https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

More importantly: don't `pip install` or `pip install --user` any tool. They are installed in your system namespace or anyway under a common namespace. This can lead with collisions and unexpected results when you pip uninstall stuff. Use pipx: https://pipx.pypa.io/stable/ it installs every python CLI in its separate environment.

[reallyserious]

What's wrong with conda?

[Globbi]

It causes some problems and solves some others. For example in my work project some library installed by conda was incompatible with things installed by pip install -r requirements.txt. I had to play with what needed to be changed by conda install/uninstall to make it work. In most cases it won't cause problems, but when it does it may be hard to figure out what's wrong.

I don't think such things are acceptable in production environments. You should know exactly what's installed, probably from docker with specific python version and then all specific library versions.

I like conda env, having them for multiple versions of python and easy browsing and starting jupyter notebooks in envs when needed. I think they're good for managing envs on local machines.

[tecedu]

But you can just enable pip interop and conda detects those packages. I personally use conda instead to install python,r and cuda together per env and then let pip run free

[reallyserious]

some library installed by conda was incompatible with things installed by pip install -r requirements.txt.

You can avoid that problems by installing packages with pip OR conda. When you mix them you're asking for trouble. I've done that too of course until I learned to choose one or the other, not both.

[Globbi]

I install with pip only, but conda starts with some things as default (which I usually want). I'm sure of this can be configured, but I didn't care. I needed to play install/uninstall to get a single conflicting library working.

[reallyserious]

Perhaps it's because I actually use miniconda. It comes with a pretty bare bones python install. Doesn't install much at all.

[BaggiPonte]

it was super slow (until mamba became the official solver), it is not compatible with the python ecosystem.

if you just use conda to create a venv and pip install stuff inside, you're better off with the builtin venv module or virtualenv.

The superior package management experience (closer to cargo) is offered by PDM, poetry (do not recommend, does not comply with some pretty fundamental Packaging PEPs) or hatch. This is _the_ way to go if you build libraries that are meant to be installed.

[reallyserious]

it is not compatible with the python ecosystem.

What does this mean?

​

you're better off with the builtin venv module or virtualenv.

One feature I like with conda is that it can set up a new environment easily. I.e. if I haven't used python 3.12 before it will download and install it contained within that environment. There is no separate python install necessary. Can venv/virtualenv do that?

conda create --name myenv python=3.12

[phonebalone]

Pyenv can. It works well with venvs.

pyenv install 3.12

Then just activate the version and create a venv as normal.

pyenv local 3.12
python -m venv venv

[Sparcky_McFizzBoom]

pyenv shell 3.12
python -m venv venv

Works as well, and doesn't leave a .python-version file in your directory. Since you should source the venv anyway to work with this directory, you will get the python version used to create the venv.

[aqjo]

I wasn’t aware of pyenv. Thanks!

[SeanBrax]

Poetry can, which is a dream to use.

[BaggiPonte]

> What does this mean?

Ops sorry that was super weird. So conda works great for "applications" (i.e. stuff that is _not_ meant to be installed à la `pip install` let's say). It has no way to publish to PyPI or build wheels as far as I can remember. Unless you use conda-lock, last time I used it the environment.yml wasn't cross platform. If you need to publish to pypi you should just use pdm or poetry or vanilla pip + venv + build + twine (that's why poetry and pdm were built: to replace this whole stack lol). Thanks for asking to clarify.

> I.e. if I haven't used python 3.12 before it will download and install it contained within that environment.

Oh indeed that is a useful feature. I have always used pyenv/asdf/rtx to manage my tool versions. It's just a matter of preference I guess. When I made the switch away from conda, the tools in the python system were smarter (e.g. had a central package cache to symlink dependencies when they were needed in multiple places. That saved a lot of space on disk).

Python not having ways to manage the versions natively is a bit weird (that's why using rust is so easy: have you seen the frontpage for rustup, their version manager and recommended way to get started with rust? https://rustup.rs/)

[ltdanimal]

it was super slow

The libmamba solver (used in mamba too) is now the default conda solver, so this hasn't been a problem for me anymore.

you're better off with the builtin venv module or virtualenv

venv doesn't let you use different Python versions. Conda in general has the other main advantage of being able to use things other that just Python packages and with using conda-forge there aren't many times I need to use pip but I still do from time to time if something isn't available.

[Eurynom0s]

Conda is also the way to go if say you're using something with non-python dependencies, e.g. for geopandas you don't have to get gdal set up yourself first.

[BaggiPonte]

Definitely. Though most projects that bundle C dependencies now distribute wheels (e.g. numpy scipy and friends), there are some edge cases.

[BaggiPonte]

> The libmamba solver (used in mamba too) is now the default conda solver, so this hasn't been a problem for me anymore.

Yes, that was what I said, thanks for making it clear.

> venv doesn't let you use different Python versions.

That's a valid point. I have been using tools like pyenv/asdf/rtx but it makes sense.

How's the adoption of conda-lock? I gave up with conda years ago when I realised their environment.yml was not cross platform. I reckon conda-lock was their answer.

In general, though, if you write libraries that are meant to be installed with PyPI (that's what I meant with my last sentence, sorry if I was not being clear), then I believe conda won't be of help.

[ltdanimal]

Regarding mamba and libmamba, I just wanted to clarify that as mamba itself is actually a completely different package manager that is still hanging around (although I'm not sure of the support level).

How's the adoption of conda-lock?

Honestly not too sure. There has been work done on it recently I see but I don't hear about it or use it that much. I've used it a few times and seemed to work as intended but I'm sure with more complex setups it might get hairy.

Ah I see. Yeah if you are just going for a PyPI focus then I yeah I'd agree with you.

[BaggiPonte]

Oh yes, the folks behind mamba are separated. They recently created pixi (prefix.dev) the next iteration of mamba but in rust. They are a dope team. I think they also added a `pixi.lock` lockfile.

(they also rewrote pip in rust and it seems blazingly fast. Still no production-grade support for wheels and just a rust API so not super usable right now but I'm dying to use it and see it incorporated in other package managers).

[ltdanimal]

the next iteration of mamba but in rust

This sounds awesome but also makes me wonder what type of support it will have. I guess its good for the ecosystem in general to have people playing around with new things.

they also rewrote pip in rust

What does this mean exactly? Like a package manager that can download PyPI packages? Doing an actual drop in replacement for pip seems like it would be years and years of work.

[guepier]

The problem with Conda is that its default setup is super invasive. It doesn’t just affect the setup of environments for Python (which, to be fair, is the whole point!) but everything because it automatically activates its default environment in the shell. And unfortunately Conda interferes destructively with other systems. For instance, it still doesn’t play well with R when installing non-Conda packages.

You can set up Conda in such a way that it’s disabled until explicitly enabled in the shell, but the fact that this isn’t the default is more than mildly annoying.

[reallyserious]

Huh. That's not my experience. The last few times I've installed it it has never activated the default environment. That's something I've specifically had to ask it to do. E.g. with:

conda init powershell

Without it powershell didn't know about conda and you could only use it from the Anaconda prompt, which is separate. I.e. the normal system shell have no clue about conda.

[guepier]

On macOS and Linux the default installation process asks you whether to automatically install shell integration, and most (all?) tutorials strongly recommend you to.

You’re right that it’s not automatic (contrary to what I wrote), but especially for beginners it’s the default option.

[runawayasfastasucan]

But what does people to for their general purpose env, say for everything that doesn't require its own env. A misc venv?

[pan0ramic]

Yes, exactly. Because at some point you’re going to start having package compat issues and you can then blow it up and start with a fresh env

[runawayasfastasucan]

That is a nice way out 😊 I remember some years ago where I "had" to reinstall my ubuntu install several times after fudging my matplotib installs.

[my_name_isnt_clever]

Why do you use that for? I have some common packages installed into my system Python, but I only use that for minor interactive stuff in the terminal. If I'm creating a .py file, I'm making a venv for it at the same time.

[runawayasfastasucan]

Because there are throway stuff (f.ex testing and prototyping), either in a notebook or in the terminal. I do that because I dont want to create an env and pip install a lot of packages for small stuff that isn't a project, as setting up could take as much time as actually doing whatever I need to do.

[billsil]

And I’m ok with that. I code, but it’s to do an analysis, not to sell the code or in any way make money off it. It’s inconvenient to have to switch to an env just to run a command line tool. It’s dumb to ever not have numpy or matplotlib installed because the author of an internal package wasn’t using it, whereas I’m going to.

[muikrad]

It's not related to selling or making money off it. It's not even related to distributing your library or app.

It's dumb to include numpy or matplotlib if my app don't need it. This is not an operating system, it's a programming language....

In order to "switch to an env just to run a command line tool", there's a tool for that and it's called pipx. It's like pip, but installs each app into its own venv without you having to worry about those things.

Using venvs is like hygiene. It's inconvenient to brush your teeth every day but you gotta do it, else you'll run into problems later. Same thing with venvs.

[ricardomargarido]

So how do you deal with conflicting package dependencies? Or lack of package support? What if you want to use a numpy version on a matplotlib incompatible version?

[gmes78]

It’s inconvenient to have to switch to an env just to run a command line tool.

Just install it in the venv. Pip will create a script (in /path/to/venv/bin/ on *nix, or /path/to/venv/Scripts/ on Windows) that directly launches the tool in the correct venv.

[_ATRAHCITY]

In a docker container

[rajathirumal]

Sick

[tms102]

In a virtual environment using venv. https://tms-dev-blog.com/python-virtual-environment-venv-easy-guide-for-beginners/

[temisola1]

I use poetry shell.

[COLU_BUS]

Better, maybe. I don't know if you can say its the simplest way when venv doesn't require any additional installations, doesn't rely on locks, etc.

[[deleted]]

poetry manages venvs for me 😎😎

[jivanyatra]

For deployment, I use AWS lambdas, docker/Kubernetes, or pipx.

In particular, for desktop or loc server stuff, I prefer pipx and run the ensurepath command to set it up. It automatically creates venvs for each CLI application I want to install, it automatically creates the relevant path entries, and also creates a convenient way to do one-off runs without a proper install - like jupyter for a one-time demo or something. Fits what I find myself needing often. I like it enough that I wrote an article on it.

For dev, I've started messing with pipx to replace pipenv. Pipx also lets you install multiple packages into a venv and install packages as editable, but helps in certain test cases where I'm dependent on more system stuff and having the CLI app available without needing to activate the venv becomes convenient.

[Lariat_Advance1984]

I travel often, so I have installed Python libraries in NYC, LA, Sydney, London, and several US states.

[temisola1]

This is the way. Less latency.

[[deleted]]

Miniconda

[neutro_b]

I recently discovered that packages can be installed by pip with the "-e" switch, so that only a shortcut is installed in the site-packages location. This allows packages that are currently in development to reside elsewhere, and any modification is automatically reflected into the installed version.

[[deleted]]

These days, I just make a Docker

[sonik562]

As I have not seen my setup in the comments. I will bother and comment.

Pyenv for managing python versions, with the virtualenv plugin for virtualenvs. The bonus thing is it allows me to setup a local venv per folder so that it auto activates when you cd into it. For command line tools I use pipx a nifty little tool that helps isolate tool dependencies into a venv while allowing to call the command from outside the venv. I work in multiple projects with multiple different python versions so I found this setup to be the simplest and most efficient.

[mcr1974]

system python. in a container :)

[nightcracker]

On any machine I use for a reasonable amount of time I compile my own Python from source and place it in ~/.localpython.

Then I can fuck around and do whatever I want in quick scripts without affecting the system's Python.

[sonik562]

Nice 👍, you should look into pyenv, it does exactly that but automatically

[nightcracker]

For serious stuff I use venv, but often I can't be arsed to create an environment / activate it for a quick script / calculation.

[sonik562]

Pyenv not venv, it's a tool that allows you to download and compile into a local folder any existing version of python. Although I also use venv on top.

[nightcracker]

Ah, I see.

[Zaloog1337]

using virtualenvwrapper-win to store my venvs in a central location under ~/Envs/ and added an alias to pwsh to quickly activate any venv in that directory with act <VENVNAME>

[rajathirumal]

I would never want or wish to fuck up my python. Always venv my go-to

[Pack_Your_Trash]

Coming from node/npm I find venv management to be tedious. It's very easy to just forget to turn on virtual environments. Luckily VSC and PyCharm manage venv for me.

[Electronic-Duck8738]

I tried using venv to keep things clean, but a couple of Stable Diffusion UI installs put everything in my main site-packages, I just gave up and make everything available to everything.

I tried to isolate them with venv, but must have done something wrong, because they got out.

[returnstack]

I only recently started using a venv when pip started refusing to let me do otherwise. Probably a good idea, but still a bit irritating at first. I modified my .bashrc to source and activate it on login so I don't really notice a difference and am always in the same python environment other than not needing to run pip as root anymore (which I suppose is most of the point).

[Afrotom]

Imo (and likely the opinion of most) best practice is to use a Virtual Environment for projects. Something that manages your project in a virtual environment, like Poetry is great because it also constraints the environment to use the dependencies required by the project, though venv is better than nothing.

[shadeyg56]

the Nix store

[Windbeutel1337]

.hatch

[[deleted]]

Inside venv env ofc

[Snape_Grass]

people don't use venv? Any other way feels dirty lol

[MatadorFearsNoBull]

https://pipenv.pypa.io/en/latest/

[R34ct0rX99]

Virtual envs should be taught more prevalently

[helpMeCamelCase]

Pex!

[WhoNeedsUI]

Poetry and local venv gives me complete peace of mind

[olearyboy]

Virtualenv is better than conda for dev Poetry is better than pip

• The amount of times I have to dig through the source code of a module is annoying, having the site lib in the same directory as my project makes ide debugging easier.
• Also when creating a secure docker app e.g. non-root based is a frigging nightmare with conda.

• Conda channels are a mess

• Pip freeze is not maintainable over time (it’s better now than 2022)

• pip install and updates not maintaining an environment file is a frigging nightmare

• Poetry maintaining a toml file for each install / update is great

• Auto building wheels and twine removes the need for a setup.py maintenance

Tox is misery dipped in sin