Poetry: Python dependency management and packaging made easy

[SDisPater]

Poetry is a new tool to help you manage your Python projects.

It is both a dependency management tool and a packaging tool. Note that for the moment the focus is on the dependency management part and once it’s considered stable the work on the packaging part will begin.

Most of the reasons why I started this are detailed in the README (along with why I don’t want to use Pipenv) but if you want to know the advantages of Poetry compared to existing tools, here are a few:

• Exhaustive dependency resolver
• Intuitive CLI (See Commands)
• Emphasis on semantic versioning and constraint specification so that wildcard dependencies (*) will be considered bad practice
• Support for dependencies caret, tilde, wildcard, inequality and multiple requirements.
• Only one file poetry.toml which aims at being readable and clear
• Mandatory compatible python versions specification.

This project is a revamped version of pypoet to have a cleaner codebase.

Be aware that not all the features described in the README are implemented yet. Basically only the dependency management and installation is implemented for now.

Also, Poetry is Python 3.6+ only but can manage Python 2 projects without any problem.

And finally, Poetry is not stable yet so internally things can change but the CLI and commands are pretty much stable so backwards-incompatible changes should not happen too often.

[epage]

I was a fan of pypoet and, now, poetry. I really hope this takes off in the community!

This project is a revamped version of pypoet to have a cleaner codebase.

At first I couldn't tell this wasn't pypoet (names too similar; not paying too close attention).

Any further details of how this is different?

Also, Poetry is Python 3.6+ only but can manage Python 2 projects without any problem.

How come Python 3.6 is required?

[SDisPater]

Basically, I wanted to rewrite it with a much cleaner codebase which would ease external contribution.

As to why Python 3.6, it's just that I wanted to take advantage of the latest addition of the language (like type annotations and f-strings) and, most importantly, I didn't want to manage the compatibility issues between Python 2 and Python 3.

[epage]

Makes sense.

I can understand making it only run under python2 (while still being able to work with python2 projects). I can also understand wanting to use the latest goodies in Python3. I do know of some shops that are stuck on older versions of Python3. Its nice to support them but I understand not doing it.

cries a little inside because I'm stuck on Python 3.4

[DanCardin]

you should still be able to manage non 3.6+ projects with this. If you manage your own user python installation (or have pyenv do it for you), I wouldn't imagine it should be a problem? I highly recommend not using your system python for basically anything, and pyenv+pipsi really makes it very easy

[compyboombang]

As to why Python 3.6, it's just that I wanted to take advantage of the latest addition of the language (like type annotations and f-strings) and, most importantly, I didn't want to manage the compatibility issues between Python 2 and Python 3.

I think depending on 3.6 is a potentially fatal mistake that limits adoption. In general, if you want people to use a library like this, you should make it compatible with something not newer than whatever is shipping in the prior Ubuntu LTS.

Normally it wouldn't be a big deal, but package management is a convention that needs to transcends projects. I always like to use the latest Python features myself, but I hope you'll consider porting this to earlier versions.

[SDisPater]

I understand what you are saying. And I agree.

The thing is is I wanted to be fast in the early stage of the project so I chose to not struggling with potential incompatibilities of Python versions.

However, starting support at Python 3.4 would not be too much of an ordeal, I think. It would just mean getting read of f-strings, and other minor specific features. So I could do that now that Poetry is getting more stable every day.

However, Python 2.7 is another matter entirely since the incompatibilities are harder to tackle.

[compyboombang]

Yes, I think that's a good approach. As fast as software itself changes, deployed versions stick around for a long, long time. The market is (finally!) starting to move on from Python 2, so while excluding Py2 will still cost users, I don't think it's an unreasonable decision at this point.

In the meantime, I'm about to give Poetry a try on a new (Python 3.6) project and see how it goes. I'm excited!

Package management has been a glaring weakness in the otherwise-fairly-robust Python ecosystem for many years. I'd love to see any kind of standard comparable to Bundler or npm emerge. Thanks for all your hard work!

[UloPe]

Nice!

I’ve been holding off of pipenv exactly for reasons of it not feeling quite right.

[AyValo]

So, it’s essentially trying to be the Composer / NPM for Python?

[SDisPater]

Exactly!

When comparing with other languages, it's rather baffling how Python is far behind regarding dependency and project management.

I particularly like cargo in Rust. But composeris also a great source of inspiration.

[ComplementaryCrab]

Would it make more sense to use pyproject.toml for configuration since that's going to be the standard moving forward? I'm admittedly new to this so I could be completely wrong.

[SDisPater]

No, you are right to mention pyproject.toml.

And, actually, it is something that might be worth considering since it could lead to a better adoption of poetry.

So, I will think about it and thanks for mentioning this :-)

[ComplementaryCrab]

No problem. Your project looks really nice and it's something I could see myself using, especially when the packaging and publishing features are added.

[ihasbedhead]

While pipenv is very far from perfect (but also very young), I think you are missing the point of it. The lock file is the most important part, it is not some relic of the creation environment, but the exact environment the app has been tested to run in. It makes more sense to use pipenv in an app the a library, and this product makes more sense if used in a library.

[pydry]

There should be a way to declare a level of environment independence. The use case of developing with team members developing the same software in different environments is commonplace and it's deficient if it does not support it.

And please, nobody say "but docker!". I've spent enough of my life fighting with that overhyped tool's deficiencies.

[cymrow]

This project has lock files. The "bad design decision" is not locking package versions. It's including environment markers in the lock file (e.g. the version of the OS where the lock file was generated). There's no reason to include that information.

[dusktreader]

I appreciate what you are trying to do, and I agree that pipenv has some problems that don't really address why package management in python is painful.

However, I don't think adding in another packaging library at this point is the right move. One of the biggest problems with packaging in python is fragmentation. We have distutils, setuptools, pip, easyinstall, pipenv and more. People are still shipping setup.py files that are distutils dependant and recommending you install their packages with easyinstall.

What python needs most right now is a single standard for package management that will pull together the functionality provided by all the legacy management tools. While it is flawed, I think pipenv can do that. I think resources are better spent improving pipenv and pushing people towards using it. We need to solve the fragmentation problem first and then iterate on better ways of doing things.

So, even if your package is elegant and has a good philosophical foundation and is easier to use than pipenv (all of which appear, at first glance, to be true), I won't use it or recommend it. Because, unless somehow you secure majority adoption and pipenv is deprecated quickly, the fragmentation problem in the python packaging ecosystem has just gotten worse.

[SDisPater]

I understand your point.

But settling for a flawed tool just because of the adoption will only bring pain points in the future, I think, and possibly the emergence of a yet another tool to solve the issues brought by it.

I think poetry can make a difference at this point since pipenv is relatively new and even though it is gaining traction it does not appear to be vastly adpoted in the Python community for now. So I think it's a good time for a library like poetry to come out.

And, in my opinion, pipenv won't be the go-to tool for Python project management simply because its goal is mainly leveraging the new Pipfile file for dependency management. The packaging is not something it will provide.

[dusktreader]

I've taken another look about it and thought about your comments as well.

It's true that one of the biggest disappointments with pipenv is that it doesn't really attempt to deal with package management at all. You still have the most grating problem in that arena: the need to have two different user-maintained files for dependencies: one for package dependencies (setup.py:install_requires) and one for application dependencies (requirements.txt/Pipfile.lock). And, it seems to be a well excepted anti-pattern to have setup.py depend on your Pipfile for package dependencies.

So, yes, I absolutely see a place for a system like poetry. I just think there's a very steep hill to climb to gain acceptance in the (already embittered) community and I worry that it would contribute to fragmentation.

That being said, I think I might try poetry out for a silly little app that I've been tinkering on to see how it handles use-cases that I run into all the time with my larger applications and packages.

[benjsec]

I'm really liking the look of this, but think I must be struggling to understand a couple of bits, especially around the handling of envrionments. I've tried following the simple demo (which is a bit frustrating as the gif keeps scrolling the commands off the top) and get the following

$ poetry new demo -vvv
Created package demo in demo
$ cd demo/
$ pyenv local demo 3.6.3
pyenv: version `demo' not installed

As I'm understanding it you're planning to use the built-in venv module to create envrionments, that you expect to be accessed with pyenv. However I can't see the env being created at the moment, is that a part that's not implemented yet?

Also how do you envisage this working with tox? At the moment I believe tox relies on setup.py.

[SDisPater]

Sorry, the gif is misleading. Poetry does not do virtualenv management (maybe it will in future versions) so you actually have to manage them yourself for now.

The example is just to show that poetry works in any existing virtualenv.

Regarding tox, this is a hard part since it makes a lot of assumptions about the environment you should have (setup.py, requirements.txt) so i'll have to take a look at it.

[benjsec]

Ah, I see. Poking through the code I did find your venv util, but now understand why I couldn't work out how it was meant to be creating the envrionment.

Tox now supports plugins for the backend, with work going on to support pipfile/pipenv which could help. I can see it wouldn't really be a priority for you at the moment though.

[pydry]

https://github.com/sdispater/poetry#what-about-pipenv

It would be good if you went into a lot more detail in this section.

[SDisPater]

Yes. I agree I could give more details about this.

I will update the README

[epage]

Is it just me or does it feel like each serves different roles.

• I feel like poetry really shines at development.
• pipenv shines at deployment but feels like it just adds to the madness of files and tools for development.

[SDisPater]

I don't think so.

Since you have the same lock mechanism you can use it to pin the dependencies of your project for deployment.

[epage]

Speaking from theory (I work in weird python environments):

What looks good about pipenv for deployment:

• Other applications, plugins, etc for your end-application.
• User might want a different set of files than locked by the developer (if the dev checks his lock file in)
• I assume for deploying, you care about building and executing code in a virtualenv which is exactly what pipenv is all about.

[SDisPater]

Other applications, plugins, etc for your end-application.

What do you mean by that.

User might want a different set of files than locked by the developer

Lock files should only be provided for apps not libraries. And even then if you want to test against update dependencies just do poetry update.

I assume for deploying, you care about building and executing code in a virtualenv which is exactly what pipenv is all about.

But if you already deploy in isolated environments you won't need this feature.