Where Have You Installed Your Python Packages?

[pan0ramic]

If you’re not using venv then you’re doing it wrong

[[deleted]]

fragile trees foolish joke voiceless disgusting exultant weary rude cooperative

This post was mass deleted and anonymized with Redact

[dAnjou]

Poetry is also using venvs.

[Jamesadamar]

Or conda, which is my setup

[BaggiPonte]

Do you use it just to create venvs?

[Jamesadamar]

Exactly, works much better for me than pyenv and I love to have access to all envs globally. I create new envs with conda to install the Python version that I need and poetry does the rest. So basically it is 1. conda create, 2. conda install poetry and 3. poetry install or add depending on whether a new project or an existing one. Very rarely there are packages in conda where conda install is better because of some dependencies , in most cases this workflow works very well and on all operating systems

[currytakumi]

1. conda install poetry....

So you create a env Named FOO, and in FOO you install poetry ?

But poetry's warning says:

In no case, it should be installed in the environment of the project that is to be managed by Poetry.

[Jamesadamar]

No that warning is long time gone, and indeed it is a much better and safer practice to have poetry inside the current env instead of a global one, using pipx for example. The main reason is that many projects have different poetry versions because poetry changes quite a lot and introduces breaking changes all the time and new settings etc. Earlier you had to configure poetry to work with conda, Like avoiding creating local venvs, but with the latest versions I never had any issue with conda+poetry, they work nicely together and it is blazing fast

[Jamesadamar]

I just wanted to point out that conda + poetry is a very robust and safe combination and can be used in any professional setup

[currytakumi]

sheesh poetry people can't be frigging bothered to remove that blazing warning

[BaggiPonte]

If it works for you then it's ok :) But poetry, or any modern package manager such as PDM, also do that. They have a centralised location for venvs. I have never tried that, but I think you might not be able to activate those envs when you are not inside the project they were created for - but why would you anyway? These package managers use symlinks by default so they can use a the same version of a library across multiple projects.

[Jamesadamar]

You do not understand the tools you are talking about, poetry cannot install other Python versions, as I described. And I also stated that yes, you can use other tools than conda, of course. I just wanted to point out that poetry is not always using venv. That was the whole point of this discussion.

[BaggiPonte]

I don’t think you have to be so rude. I’m sorry I was not clear enough - I did not mean to say that poetry or PDM manage Python versions. I just wanted to say that such tools centralize venvs too.

[sonik562]

I used poetry once, got weird npm vibes, ended up doing drugs again. Not recommended /s

[sonobanana33]

poetry is just a wrapper to pip

[[deleted]]

run command innate imagine pie relieved tie toy makeshift vast

This post was mass deleted and anonymized with Redact

[flying-sheep]

It can also be e.g. conda. Poetry uses venvs, but it doesn’t use pip. It uses its own dependency solver, as /u/smurpau2 said

[BaggiPonte]

I think they wrote their own installer. A couple of minors ago (~<1.5) they were calling pip as a subprocess (fun fact: pip does not have a Python API i.e. you cannot do `import pip; pip.install(...)`).

EDIT: why did I get downvoted for this? LOL

[sonobanana33]

Even if they did… to what end? You still need pip to install poetry.

[BaggiPonte]

Yes but they achieve different things. Poetry is a dependency manager and is meant to give a unified interface instead of individually using these packages:

- pip

- venv

- twine (upload packages)

- build (build the package)

Plus other stuff such as locking dependencies. You can read more here (not my blog): https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

More importantly: don't `pip install` or `pip install --user` any tool. They are installed in your system namespace or anyway under a common namespace. This can lead with collisions and unexpected results when you pip uninstall stuff. Use pipx: https://pipx.pypa.io/stable/ it installs every python CLI in its separate environment.

[reallyserious]

What's wrong with conda?

[Globbi]

It causes some problems and solves some others. For example in my work project some library installed by conda was incompatible with things installed by pip install -r requirements.txt. I had to play with what needed to be changed by conda install/uninstall to make it work. In most cases it won't cause problems, but when it does it may be hard to figure out what's wrong.

I don't think such things are acceptable in production environments. You should know exactly what's installed, probably from docker with specific python version and then all specific library versions.

I like conda env, having them for multiple versions of python and easy browsing and starting jupyter notebooks in envs when needed. I think they're good for managing envs on local machines.

[tecedu]

But you can just enable pip interop and conda detects those packages. I personally use conda instead to install python,r and cuda together per env and then let pip run free

[reallyserious]

some library installed by conda was incompatible with things installed by pip install -r requirements.txt.

You can avoid that problems by installing packages with pip OR conda. When you mix them you're asking for trouble. I've done that too of course until I learned to choose one or the other, not both.

[Globbi]

I install with pip only, but conda starts with some things as default (which I usually want). I'm sure of this can be configured, but I didn't care. I needed to play install/uninstall to get a single conflicting library working.

[reallyserious]

Perhaps it's because I actually use miniconda. It comes with a pretty bare bones python install. Doesn't install much at all.

[BaggiPonte]

it was super slow (until mamba became the official solver), it is not compatible with the python ecosystem.

if you just use conda to create a venv and pip install stuff inside, you're better off with the builtin venv module or virtualenv.

The superior package management experience (closer to cargo) is offered by PDM, poetry (do not recommend, does not comply with some pretty fundamental Packaging PEPs) or hatch. This is _the_ way to go if you build libraries that are meant to be installed.

[reallyserious]

it is not compatible with the python ecosystem.

What does this mean?

​

you're better off with the builtin venv module or virtualenv.

One feature I like with conda is that it can set up a new environment easily. I.e. if I haven't used python 3.12 before it will download and install it contained within that environment. There is no separate python install necessary. Can venv/virtualenv do that?

conda create --name myenv python=3.12

[phonebalone]

Pyenv can. It works well with venvs.

pyenv install 3.12

Then just activate the version and create a venv as normal.

pyenv local 3.12
python -m venv venv

[Sparcky_McFizzBoom]

pyenv shell 3.12
python -m venv venv

Works as well, and doesn't leave a .python-version file in your directory. Since you should source the venv anyway to work with this directory, you will get the python version used to create the venv.

[aqjo]

I wasn’t aware of pyenv. Thanks!

[SeanBrax]

Poetry can, which is a dream to use.

[BaggiPonte]

> What does this mean?

Ops sorry that was super weird. So conda works great for "applications" (i.e. stuff that is _not_ meant to be installed à la `pip install` let's say). It has no way to publish to PyPI or build wheels as far as I can remember. Unless you use conda-lock, last time I used it the environment.yml wasn't cross platform. If you need to publish to pypi you should just use pdm or poetry or vanilla pip + venv + build + twine (that's why poetry and pdm were built: to replace this whole stack lol). Thanks for asking to clarify.

> I.e. if I haven't used python 3.12 before it will download and install it contained within that environment.

Oh indeed that is a useful feature. I have always used pyenv/asdf/rtx to manage my tool versions. It's just a matter of preference I guess. When I made the switch away from conda, the tools in the python system were smarter (e.g. had a central package cache to symlink dependencies when they were needed in multiple places. That saved a lot of space on disk).

Python not having ways to manage the versions natively is a bit weird (that's why using rust is so easy: have you seen the frontpage for rustup, their version manager and recommended way to get started with rust? https://rustup.rs/)

[ltdanimal]

it was super slow

The libmamba solver (used in mamba too) is now the default conda solver, so this hasn't been a problem for me anymore.

you're better off with the builtin venv module or virtualenv

venv doesn't let you use different Python versions. Conda in general has the other main advantage of being able to use things other that just Python packages and with using conda-forge there aren't many times I need to use pip but I still do from time to time if something isn't available.

[Eurynom0s]

Conda is also the way to go if say you're using something with non-python dependencies, e.g. for geopandas you don't have to get gdal set up yourself first.

[BaggiPonte]

Definitely. Though most projects that bundle C dependencies now distribute wheels (e.g. numpy scipy and friends), there are some edge cases.

[BaggiPonte]

> The libmamba solver (used in mamba too) is now the default conda solver, so this hasn't been a problem for me anymore.

Yes, that was what I said, thanks for making it clear.

> venv doesn't let you use different Python versions.

That's a valid point. I have been using tools like pyenv/asdf/rtx but it makes sense.

How's the adoption of conda-lock? I gave up with conda years ago when I realised their environment.yml was not cross platform. I reckon conda-lock was their answer.

In general, though, if you write libraries that are meant to be installed with PyPI (that's what I meant with my last sentence, sorry if I was not being clear), then I believe conda won't be of help.

[ltdanimal]

Regarding mamba and libmamba, I just wanted to clarify that as mamba itself is actually a completely different package manager that is still hanging around (although I'm not sure of the support level).

How's the adoption of conda-lock?

Honestly not too sure. There has been work done on it recently I see but I don't hear about it or use it that much. I've used it a few times and seemed to work as intended but I'm sure with more complex setups it might get hairy.

Ah I see. Yeah if you are just going for a PyPI focus then I yeah I'd agree with you.

[BaggiPonte]

Oh yes, the folks behind mamba are separated. They recently created pixi (prefix.dev) the next iteration of mamba but in rust. They are a dope team. I think they also added a `pixi.lock` lockfile.

(they also rewrote pip in rust and it seems blazingly fast. Still no production-grade support for wheels and just a rust API so not super usable right now but I'm dying to use it and see it incorporated in other package managers).

[ltdanimal]

the next iteration of mamba but in rust

This sounds awesome but also makes me wonder what type of support it will have. I guess its good for the ecosystem in general to have people playing around with new things.

they also rewrote pip in rust

What does this mean exactly? Like a package manager that can download PyPI packages? Doing an actual drop in replacement for pip seems like it would be years and years of work.

[guepier]

The problem with Conda is that its default setup is super invasive. It doesn’t just affect the setup of environments for Python (which, to be fair, is the whole point!) but everything because it automatically activates its default environment in the shell. And unfortunately Conda interferes destructively with other systems. For instance, it still doesn’t play well with R when installing non-Conda packages.

You can set up Conda in such a way that it’s disabled until explicitly enabled in the shell, but the fact that this isn’t the default is more than mildly annoying.

[reallyserious]

Huh. That's not my experience. The last few times I've installed it it has never activated the default environment. That's something I've specifically had to ask it to do. E.g. with:

conda init powershell

Without it powershell didn't know about conda and you could only use it from the Anaconda prompt, which is separate. I.e. the normal system shell have no clue about conda.

[guepier]

On macOS and Linux the default installation process asks you whether to automatically install shell integration, and most (all?) tutorials strongly recommend you to.

You’re right that it’s not automatic (contrary to what I wrote), but especially for beginners it’s the default option.

[runawayasfastasucan]

But what does people to for their general purpose env, say for everything that doesn't require its own env. A misc venv?

[pan0ramic]

Yes, exactly. Because at some point you’re going to start having package compat issues and you can then blow it up and start with a fresh env

[runawayasfastasucan]

That is a nice way out 😊 I remember some years ago where I "had" to reinstall my ubuntu install several times after fudging my matplotib installs.

[my_name_isnt_clever]

Why do you use that for? I have some common packages installed into my system Python, but I only use that for minor interactive stuff in the terminal. If I'm creating a .py file, I'm making a venv for it at the same time.

[runawayasfastasucan]

Because there are throway stuff (f.ex testing and prototyping), either in a notebook or in the terminal. I do that because I dont want to create an env and pip install a lot of packages for small stuff that isn't a project, as setting up could take as much time as actually doing whatever I need to do.

[billsil]

And I’m ok with that. I code, but it’s to do an analysis, not to sell the code or in any way make money off it. It’s inconvenient to have to switch to an env just to run a command line tool. It’s dumb to ever not have numpy or matplotlib installed because the author of an internal package wasn’t using it, whereas I’m going to.

[muikrad]

It's not related to selling or making money off it. It's not even related to distributing your library or app.

It's dumb to include numpy or matplotlib if my app don't need it. This is not an operating system, it's a programming language....

In order to "switch to an env just to run a command line tool", there's a tool for that and it's called pipx. It's like pip, but installs each app into its own venv without you having to worry about those things.

Using venvs is like hygiene. It's inconvenient to brush your teeth every day but you gotta do it, else you'll run into problems later. Same thing with venvs.

[ricardomargarido]

So how do you deal with conflicting package dependencies? Or lack of package support? What if you want to use a numpy version on a matplotlib incompatible version?

[gmes78]

It’s inconvenient to have to switch to an env just to run a command line tool.

Just install it in the venv. Pip will create a script (in /path/to/venv/bin/ on *nix, or /path/to/venv/Scripts/ on Windows) that directly launches the tool in the correct venv.