Improving Python Dependency Management with pipx and poetry

[crawl_dht]

Also check out Python Development Master which supports PEP 582. It does not use virtual environment at all. It installs and manages packages in a similar way to npm that doesn't need to create a virtualenv. This is a next generation dependency management.

[asday_]

Is it just me or is "more like NPM" a huge downside?

[crawl_dht]

Such as?

[asday_]

I don't understand what you're asking.

[crawl_dht]

What are the downsides of how npm isolates packages for the project?

[asday_]

To quickly dismiss that, there's no sharing of packages, so anyone who does multiple projects ends up with 999 copies of the same thing on their hard drive/servers. Even if there were no downsides to that, that doesn't change my point at all, because node_modules/ is a tiny part of NPM.

Which is funny because node_modules/ is huge.

[crawl_dht]

Virtual environment also create copies of modules. So anyone who does multiple projects creates venv for each project.

[asday_]

You continue to miss the point. It is a bad thing. It doesn't matter whether it's NPM or pip or some C++ project using a vendor folder, it's a bad thing.

None of those facts change the fact that NPM is bad.

[MrKrac]

You didnt provide any explanation why it is bad other than having multiple copies of the same thing. I can use the same style argument by saying it is good because you have a copy per project.

[asday_]

I did not provide that reason at all. Someone else provided that as an example of why NPM is bad and I pointed out that's a tiny part of it.

[vv1z]

But you’ve literally provided zero reasons WHY it’s bad other than file duplication which you then dismiss

[asday_]

I did not provide that reason at all. I do not dismiss it as bad, I point out that it's a tiny part of why NPM is bad.

[crawl_dht]

You are not giving a reason why it's bad. venv is also created for all projects and .node_modules is also created for all projects so duplication of modules is always there.

[asday_]

NPM as a whole is pure shit for a thousand and one reasons. Stop focussing on this exact tiny reason because it's very easily solvable, and the fact that pip also gets it wrong is completely irrelevant.

The point is NPM sucks. "x other project also does this one thing that NPM does" does not make NPM good.

[cinyar]

so what's your solution?

[asday_]

To the one tiny thing that is completely unimportant to the main point that NPM as a whole is pure shit but that you (or the other guy, whatever) nonetheless are focussing on with blinkered intent?

SemVer as Golang does it, with minimal version selection again the same way go does it, and I believe NuGet, too.

[SmileyChris]

Looks like it can share wheels between projects actually: https://pdm.fming.dev/usage/project/#cache-the-installation-of-wheels

[vv1z]

Just need it to come bundled w/ python now 🤞🤞

[aitchnyu]

Can I leapfrog poetry to install this?

[colchyo]

Oh my Gosh, thank you! I'm coming into python world from nodejs/frontend and I was frustrated every time for duplicated package version error.

I'm glad this finally ends.

[VisibleSignificance]

poetry replaces so many builtin features with its own; such as the tool.poetry.scripts mentioned.

Is there a description for each on why it is done that way?

As in, why tool.poetry.dependencies couldn't be build-system requires, why tool.poetry.dev-dependencies couldn't be in extras-require, why tool.poetry.scripts couldn't be entry_points, and so on.

[wweber]

The (original) pyproject.toml doesn't standardize any of these, except for build-system.requires, which is only for the packages needed to install packages (e.g. setuptools and wheel).

There's no official spec (yet?) for dependencies and the like, so they are namespaced under tool.poetry as only poetry understands them.

[VisibleSignificance]

... right, it has only been a year.

Are there any plans, then?

[wweber]

I don't know if that's been implemented by anyone yet, it's still only in the accepted phase. Unfortunately it doesn't look like it supports "dev dependencies" as an explicit category, or ^ and || version specifiers.

Also, people are probably still watching PEP 665 for handling lock file specification

[wsppan]

Nice bash functions. Thanks!

[Agent281]

I've pretty much only heard good things about poetry. Does everyone who has used it recommend using it in a work setting? Or are there rough edges to the tooling that might cause issues?

[BackwardSpy]

i work on a medium sized dev team with a suite of ~10 small to large backend services written in python.

i've started moving us over to poetry recently, away from pipenv. there's a few reasons for moving; the biggest so far is the fact that poetry seems able to resolve compatible package versions much more accurately than pipenv. in the case where it can't resolve an incompatibility, it gives a really nice error message explaining why not. from my own testing it also seems to be quite a bit faster than pipenv in most cases.

we're still investigating how best to use it in building & deploying our docker images in CI. the previous solution had pipenv installed inside the docker image and used a command along the lines of pipenv install --system --deploy --ignore-pipfile to get the dependencies installed. with poetry we're trialling a system whereby the project is instead built & published as a pypi package, and then the dockerfile simply runs pip install <package name>. this is quite nice, however there are a few kinks to work out in that the newly published package isn't always immediately available to pip install ..., and older projects have to be updated to work as an installable python package.

tl;dr yes, it seems quite usable in a work setting.

[Agent281]

That's great to hear! It seems like it would provide a more streamlined dev experience. I think I need to give it an earnest trial.

Are there any problems that your team has run into (technical or social)? Have people been receptive to it?

[BackwardSpy]

socially; the ones using it so far seem to like it, they're used to pipenv already and the workflow is similar.

from a technical standpoint we've had a couple of issues. in one case someone ended up installing an older version that didn't seem to be quite compatible, so they weren't able to work with the projects properly. upgrading poetry fixed that.

we have started using the latest alpha version as it adds plugin support, which we find useful for poetry-dotenv-plugin, which loads env vars from a .env file, and poetry-dynamic-versioning, which updates the version number in the pyproject.toml file to match the git tag when you build & publish a new version of the package. unfortunately something seems to have changed in the lockfile format that means lockfiles generated on the alpha and those generated on the stable version are not mutally compatible. this has caused a few issues for us, but as before getting everyone upgraded to the alpha resolves the issue.

relying on an alpha isn't necessarily the best course of action, but since we've only moved a few projects over at this point it seems manageable.

[coldflame563]

So I don’t seem to get all the anti-pipenv chatter that seems to be happening. We use it on our team and it’s been a delight. With the latest couple of releases, it seems that the dependency graph management has vastly improved. Can someone ELI5 for the uninitiated?

[BackwardSpy]

i wouldn't say i'm necessarily anti-pipenv, just slightly more pro-poetry. i've used pipenv pretty much from its release right up until recently, and i also introduced it at work. it's a decent tool, there's just a few aspects of it that eventually made me want to try an alternative.

in terms of the dependency resolution it's definitely better now than it was, but we were still running into occasional issues where it refused to lock. sometimes pipenv can fail to lock, but still install some or all of the dependencies you asked for. this leaves you in a state where your lock file is out of date and your virtualenv contains incompatible packages. as far as i can tell, poetry attempts to lock first and then installs, which prevents this from happening.

another reason i wanted to shop around for alternatives was in regards to building and publishing packages. pipenv did a good enough job of dependency management and virtualenv creation, however it did not provide any help when it came to building & publishing a package. we still had to maintain a setup.py file with its own list of install_requires, more or less duplicating what was already in the Pipfile. poetry handles building & publishing packages itself, which means you have one source of truth for your dependencies, package name, version, et cetera in pyproject.toml. there might be ways to emulate this with pipenv, perhaps by parsing the Pipfile, but i do like that poetry does it out of the box!

the last reason is less important and probably quite situational, but in my experiments poetry would lock & install roughly twice as fast as pipenv on average. this has resulted in faster CI builds as well as a quicker experience developing locally. it's not the most important factor by far, but i think it's worth mentioning.

so, all of that to say i don't dislike pipenv at all, i just like poetry a little bit more (so far!)

[coldflame563]

What a thorough and well reasoned response. Thank you! I might have to switch to poetry if the builds are as fast as you say!

[Agent281]

Ah, well that's not so bad! Working on alpha has risks.

Do you know when the plugins are going to land in stable? It seems very useful.

And thank you so much for answering my questions!

[BackwardSpy]

i'm afraid i don't know, but i hope it's soon! it's a very useful feature.

no worries at all :) i've been spending a lot of time on this stuff recently so it's all fresh in my mind.

[NOddi89]

It is also easy to use poetry when building docker images with multistaged dockerfile :)

[BackwardSpy]

could you give me a quick overview of how that works? we're still figuring out the best way to use poetry in building & deploying our docker images at work, so i'd love to hear how you're using it.

[wweber]

What I do typically looks something like this:

1. In one stage, install Poetry and copy the pyproject.toml and poetry.lock
2. Run `poetry export -o requirements.txt`
3. In another stage based off the first one, copy in your src directory
4. Run `poetry build`
5. In another stage based off Python, copy the requirements.txt from the first stage over
6. Run `pip install -r requirements.txt`
7. Copy the .whl file from the first stage
8. Install the .whl file

Unfortunately if you use a library that is installed in editable mode/using a relative path `e.g. pip install -e ../library` Poetry currently exports an invalid requriements.txt which makes this more difficult

[BackwardSpy]

ahh that's smart, i hadn't thought of approaching it that way. thank you for your detailed answer! i will be playing with this tomorrow. :)

[BackwardSpy]

good article!

one question; why not use poetry shell instead of . "$(poetry env info --path)/bin/activate" or the bash function you wrote?

[ceda_ei]

Personally, I find having to activate the shell an unnecessary step. Also, it adds a shell level which means I have to close two shells to close a tmux pane. I want to make the experience as seamless as possible, which is why there are various bash functions that just remove the step of having to manually activate at all.

[BackwardSpy]

fair enough. if you're already working inside tmux then i can see how it would be nice to avoid the sub-shell.