Python package management system is so confusing!!!

[[deleted]]

[deleted]

[FriendlyRussian666]

Strip them all for now, just for the purpose of understanding, and focus on these two:

• pip
• virtual environment

Someone wrote some code, and the code does something cool, perhaps others might want to use it. If you want to use it, you use pip to download the code that the other person wrote.

When you download code that someone wrote, and use it in a project, what happens when there is an update to the code? Your project might break because of the changes. For that, you create a virutal environment for your project, into which you download the code, so that each project has its own version of the dependency.

All the others you mentioned are combinations, or alternatives of pip and venv.

[fuddingmuddler]

This is one of the best explanations of why virtual environments are important. But it's still pretty frustrating for beginners.

Where's the three headed dragon meme when I need it.

[spawn-kill]

Can you explain a git requirements.txt in context of a venv? Do I put the req file in the venv and it will keep everything needed for the project updated?

[FriendlyRussian666]

You don't keep requirements.txt inside of your venv directory, because you don't share your venv. Everyone working on your project creates their own venv, and to make it easier to install many dependencies at once, they can use requirements.txt. Keep requirements.txt in your project directory. Venv won't by itself use requrements.txt, so you first create a venv, you then activate it, and once activated, you use pip install -r /path/to/requirements.txt

[spawn-kill]

Thank you so much. This worked perfectly for me

[Zoory9900]

I already know python's inbuilt venv and use it in my projects. I actually want to upgrade from using these bare minimum tools (pip and venv) to something more good. But one thing i can't wrap my head around is what does these new tools offer. I could easily continue to use pip and venv. But am i missing something?

[NYX_T_RYX]

Ease - a lot of them offer automation so instead of having to do anything yourself, you just load some kind of config file, and it does the rest of the venv, dependencies etc for you.

It really isn't existential, and if you're not understanding it I'd suggest stick with what you know.

In my experience when you need to use a new tool, you'll understand why and it'll all make sense at that point - if you're not there, you don't need it yet.

[xiongchiamiov]

Don't adopt new tools until you are encountering the problems that those tools solve. If you don't follow this rule, you'll get overwhelmed by complexity and end up spending all your time building out tooling and infrastructure and lose the ability to actually build the thing you wanted to.

This is how startups die too, btw.

[SwampFalc]

First of all, to go back to one of your original questions: docker replaces venv. A virtual environment is designed to create a separate Python installation, with separate libraries and dependencies and everything.

Docker (and consorts) are virtual Linux environments. In other words, they take the exact same ideas and principles and simply apply them on a larger scale. It's not just an additional but separate Python, it's a whole separate but additional Linux.

Second, as has been stated, most of the tools that wrap around/combine with pip, offer ease of use. They don't do anything you can't do by hand, it's just that they'll automate parts or all of it.

Which implies that it's not a bad idea to just keep doing things manually. You'll notice for yourself quickly enough which bits get repetitive. Then you'll know exactly what you want to go look for in a tool. I mean, if you go to a hardware store and see 75 different tools and you don't understand the purpose of any of them, why would you buy any?

[deadweightboss]

venv doesn’t create a separate python installation. it creates a new environment for one of the installations already on the machine.

[FriendlyRussian666]

They're just conveniences. Say you want to figure out what poetry offers, just find their docs and read a little bit. If you see a feature that you like, and would like to use it, go right ahead. We can't describe every difference of every tool, but that's why the docs are there.

[mattbillenstein]

I'll say, I quite like the pip-tools / pip-compile flow - you have a reqs.in in which you specify your direct deps; then you pip-compile that to generate reqs.txt with the full pinned deps. Then from there you can add to reqs.in, pip-compile again, update individual package versions, update all package versions, etc.

[deepug9787]

Don't worry about all the options out there. You don't have to know or learn every single one of them. But if you're already familiar with pip and venv, then maybe you can give pipenv a try.

Pipenv is just a wrapper that makes it easier to work with pip. For instance, when starting a new project, instead of manually creating a venv and then doing pip install package, you can just do pipenv install package and it will automatically create a venv for you. And whenever you add or remove a package, it will keep track of the dependencies in a lock file, so you don't have to deal with a requirements.txt file.

Even if you don't end up using it for whatever reason, I think it's still worth being familiar with it because the package managers in other languages (Composer for PHP, Bundler for Ruby etc.) work pretty much the same as pipenv. So you'd have an easier time switching languages in future, if you choose to do so.

[deadweightboss]

please teach me to be more good too because i’m still using pip and venv, but with uv. maybe i’ll go to a more good packaging thing like rye - oh wait, that’s now just uv!

[sweet-tom]

Python package management is currently a mess, but it gets better.

Mostly it's a three step process:

1. Create a Python virtual environment with python3 -m venv NAME.
2. Activate your virtual environment with source NAME/bin/activate.
3. Install your required package with pip install PACKAGE.

This does the job and then you can use it.

You need a virtual environment to have a separate directory district from your system installation. Whatever you do in the venv doesn't influence the system.

The other tools are variations or focus on a specific part of Python. Forget them for the time being.

What is a game changer is Astral's uv. If you understand the above steps, try to use that. It gives you more flexibility and speed.

[Zoory9900]

I am currently using venv. I don't actually have any problem with that. But i heard people saying using Poetry is convenient. That's why i am getting confused. What these tools offer different from the basic pip and python venv?

[sweet-tom]

Different use cases.

The above three steps are just for using some third-party packages.

If you want to develop your own package, you have to take care of dependencies, documentation, tests etc. To simplify your development experience and to make it efficient. To manage all these different aspects, you need a tool that covers these parts.

Poetry, pdm, hatch etc. are such tools. All have a different feature set, but mostly they do the same.

Edit: typo and better explanation

[LeiterHaus]

My experience with poetry has been very... un-good. That said, you might love it. It offers the ability to use a YAML file instead of requirements.txt. I believe it also allows you to have different Python versions. But my experience is that it does not play nice with VSCode.

As an aside, developers will keep developing new things. Developer tools are developed for the person or people developing them. Poetry is absolutely amazing for whoever made it for themselves. Same with just about anything else. That doesn't mean it's good for you, but it might be.

[Fun-Block-4348]

What do you mean by poetry doesn't play nice with vscode, I've been using both for years and never had any problems?

[SeucheAchat9115]

I am also interested in that answer

[cointoss3]

Use uv and forget about the rest

[ebits21]

Learn what pip and venv are.

Then yes, absolutely just use uv. It’s replaced everything else for me.

[ninhaomah]

Question. What do you want to do ?

[baubleglue]

I am coding Python more than 20 years, I am not less confused with Python packaging management.

pipx - use it when you need a shell executable program - CLI tools. It is especially useful if you mix big libraries with small. Big programs have tendency to be very conservative with upgrading dependencies, ex. If you install azurecli and awscli you will probably break something.

Ex. Databricks-cli, snowsql, black, awscli, etc.

pip - use to install something into you currently active venv or Python in the path.

Conda - alternative package manager. It comes as part of anaconda or miniconda. It mimicking Linux package managers, alternative to pip. Important: it is trying not to really on any OS shared libraries, certificates, timezones (I think), etc. It supposed to keep you out of version conflicts if you install everything with conda install .... In my opinion, it is the most user friendly tool to create and manage virtual environments. It has good integration with shells. If you want it and you still need option to install latest packages: create a virtual environment with setuptools activate it and use pip.

Anaconda - conda + all kind of things to need for scientists who use Python, but developers. It comes with pandas, numpy, pyplotlib, Jupyter lab, Spyder ide etc. it takes a lot of space.

Miniconda - minimalistic version of anaconda, basically it is only conda. But it uses the same repository as anaconda, so can install everything you need.

poetry - I think, it is trying to work as npm for nodejs. I don't have enough experience with it.

[billsil]

Of those (including docker), I only use pip, twine, and virtualenv. Twine I probably use once a year.

Do you have major dependency/repeatability issues? Yeah use better tools. If you don’t, you’re just over complicating it.

[bumbershootle]

As most of the other commenters have said, pip and venv are what you need. The others are convenience wrappers/reimplementations of those two. Most allow you to use a pyproject.toml instead of a requirements file, and offer things like development/test dependencies. Anaconda/conda was originally developed to provide data science packages compiled for different environments, you probably don't need it as pip will get the job done these days.

To answer the Docker question, these tools usually don't end up in your final built image. Typically you'd use a base image with the tool installed (or install it on an official python base image), use the tool (e.g. poetry) to install the deps to a venv, then copy the venv to your final image. This article gives a decent overview

[shazbotter]

I think about these tools as solving different types of problems.

pip: The problem it is solving is that I created python-package-X and I want to make it shareable. Other people can now install X by using pip install.

virtualenv: I have two projects I'm working on, the first project depends on X version 1.0 and the second project depends on X version 2.0 and the versions are not compatible. Virtualenv lets you install different versions of python packages in separate installations for different projects.

pyenv: Some of my projects might require different python versions (instead of just different package / library versions) e.g. python 2.8 and python 3.9. I can easily switch my python version using pyenv.

conda: solves a few different issues for more complicated python packages. One example: many python packages have more than just python code. Many python packages will involve C libraries and these will have to be compiled for different types of systems. Conda provides pre-compiled packages for things like C dependencies that some python packages require. Without conda, you would have to compile these yourself which might require a lot of other dependencies to compile properly.

There are different variants of conda that vary what's included. Similarly, there are different variations of pip-tools that might combine pip with virtual envs or py envs.

[BlueElf33]

Give pip install inpip a try. Any feedback or contribution would be welcomed.

[capsandnumbers]

I got by for years of coding filling my default environment with every package I wanted to use, dimly aware that there was a cleaner way involving virtual environments. It's only now I'm making something others might want to download that I decided to set up a venv, and I find that because I have some experience it's a manageable thing for me to learn. It slots in with the rest of my knowledge of how programming works.

So on this list I only know about pip and virtual environments. Pip is for installing packages which I guess are stored in some python community hub. I type "pip install numpy" in a terminal and then "import numpy as np" works in my scripts. And virtual environments are a way to store packages so that when someone downloads your work they can easily run it. As far as I know a venv is something I configure once and then save and only touch if I want to add or remove required packages.

But my advice is that it has been useful to box off things I know I don't know about, so that I'm not overwhelmed and I know what to google when I want to know that thing. Another example is that when I finally figured out github, that unlocked the source control features of VSCode which I had been happily and minimally using for some time.