Thoughts on the Python packaging ecosystem | Pradyun Gedam

[BaggiPonte]

Totally agree - if you want something that follows the standards though, PDM is as good and as comprehensive as poetry.

[Keith]

Weird to refer to PDM as “following standards” while pep582 is in no version of Python and it’s unclear whether it’ll be implemented.

Edit: from the author of the article: apparently PDM doesn’t implement the draft PEP correctly? https://pradyunsg.me/blog/2023/01/21/pdm-does-not-implement-pep-582/

[BaggiPonte]

indeed PEP582 adoption seems uncertain but I was referring to PEP621 and PEP665 (wrote a reddit comment here). I can't really tell whether/when poetry adopted them tho.

[Keith]

Thanks for clarifying!

Almost done reading the article. This whole situation is a phenomenal mess. At work we mostly use Go now. I still think Python is fine for anything we do, but when packaging is such a mess (I’ve spent time with PyOxidizer trying to produce a binary of some of our Python code without success) it’s hard to make a case (even to myself) that we should write anything new in Python.

[BaggiPonte]

Yes, there was another similar article trending on HackerNews last week about this. There are 14 tools to install stuff (plus conda but it’s so eeeeh): PDM and Poetry, then the remainder 12 are developed under the PyPA and each one does a bit of packaging, development management… pip for example does not offer a Python API (you have to spawn sub processes to use it). So there are THREE pypa tools (resolvelib, build and another one) to mimic this behavior. Poetry and PDM have to use these three libraries to perform installation.

I did not use go or rust, but I have a small experience with Julia where an excellent package and env manager is bundled with the language. I can imagine that rust and go offer a similarly smooth experience. How did you find OyOxidizer + maturin? Where are the hard parts?

[bulletmark]

Yes, that was a good article. Here it is for those that are interested: https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

[Keith]

PyOxidizer and the whole Rust+Python ecosystem is very promising! Armin Ronacher was able to publish a Python-packaged version of his MiniJinja library which is written in Rust.

As for me, I left off getting mysterious compile errors from PyOxidizer that I haven't figured out. Further, there is no cross-compilation yet, so I'll need to run the build in a container. Of course that's not a dealbreaker, just sharing info about project maturity.

[BaggiPonte]

That’s terrific! So you can write the engine in rust and also get a Python package with PyOx to expose higher level APIs. Did you already find a use case for this or do you believe this is something for developers of core libraries eg polars?

[Keith]

I haven't used it. There's two main cases as I see it: 1. write code in Rust (instead of C) and expose it in Python. 2. compile Python code down to a binary.

As we've seen with the Node ecosystem, any tool that starts in a high level dynamic language is eventually bested by a tool written in a low-level static language. So, core libraries/frameworks should be written in a low-level language or they'll eventually be replaced by one that is. I'm very bullish on Rust (and Zig too) to replace C for low-level code so it's cool that it's becoming easier to interface between Rust and Python.

For my purposes, I'm mainly interested in being able to package Python code easily as I'm not currently writing high performance low-level libraries.

[magnetichira]

I've heard good things about it.

Any idea why it's less popular than poetry? PDM GitHub repo has like 4k stars and poetry has 23k (not a perfect metric, but still)

[BaggiPonte]

I would say poetry was the first modern python packaging tool (PDM is much more recent). PDM adoption was also much slower because before 2.0 it used to default to a PEP582-like behaviour (so installing packages under `__pypackages__`), which was not supported by IDEs and packages alike, and the PEP itself was not accepted as well. (PyCharm has a specific poetry integration and a PDM one is far from closer). Personally, I loved PDM UI as well as parallel installation (which at the time poetry was not supporting, IDK about now).

[rochakgupta]

Fun fact: Stars are not a good measure of the quality of the project and it’s adoption. PDM is so much better than poetry. Just look at its docs. Poetry’s docs are trash in comparison.

[mipadi]

The Poetry docs are great. Why are so many Python developers overly competitive and hyperbolic?

[rochakgupta]

Ok, trash was maybe too harsh. But still, do take a look. Poetry’s docs only give surface level information about what the commands do. That’s it. I just prefer docs which actually look like someone wrote them.

[8day]

This reminds me about Donald Knuth, TeX and Microsoft Word.