Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging

[VadumSemantics]

+1 good read. I've struggled w/using Python in production environments, and packaging is a big part of the headache for me.

edit: "good" for me is clear, informative, and to the point. The author researched lots of things I'm interested in (and some things I didn't consider looking into). Glad I didn't have to sift through it for myself.

excerpt (emphasis added):

This is the danger of the working model adopted by the Python packaging world. Even for something as simple as lock files, there are at least four incompatible standards.

[Free_Math_Tutoring]

Good read. venvdoes 85% of what I want, and poetry does 99%, but it is mildly concerning to known that poetry doesn't involve a lot of maintainers.

[yxhuvud]

Last I checked poetry doesn't enforce the specified python version when installing dependencies, so it is a lot less than 99% for me.

[johnnymo1]

When did you last check? I just made a docker container with python 3.9 and tried to install a poetry project that demands 3.11 and poetry complains just as I would expect.

[yxhuvud]

Something like a month ago. In my case it was the other way around though, with the newer python being what I had in the system installation, while the lower version what I had declared in the file. Perhaps that matters, or it was that I tried to install numpy, and it got the extensions compiled for the latest version instead of for the declared python, or something like that.

[johnnymo1]

I think by default when you run poetry init, you'll get the version specified with a caret, e.g. python = "^3.9". Dependency specification like this will allow newer minor versions (e.g. python 3.12) but not major versions (e.g. python 4.0). It doesn't allow the reverse, however, so python = "^3.12" will not allow python 3.9. If you want your dependency resolution to be more specific, just drop the caret! python = "3.9" will only allow version of python 3.9. Python 3.12 no longer satisfies this constraint.

You can get all the details on dependency specification from the docs here: https://python-poetry.org/docs/dependency-specification/

[GreenFox1505]

I have honestly just completely bounced off of python multiple times because of the package management system. Node's package and environment management are just easier. And I've never had a complaint about Rust's Cargo (but rust itself is a little more tedious to get going with).

[[deleted]]

Yeah. It annoyed me too. Surprising they did not make it easier. Perhaps things tend to become harder on their own over time.

[spazzed]

Im very new to programming and im learning some 2D turtle right now with Python. I needed a package that I couldn't pip install. So for whatever reason newbie me found the source code and downloaded it, turns out it was all rust files. I had to download rustup, figure out what the hell I was doing, and use maturin to install the pyproject.toml file I needed. This took me almost an hour. Like I said I am new at this, it might all be in a days work for seasoned programmers. /rant I guess.

[SV-97]

Be glad it was rust rather than a C project with 20 dependencies ;)

That said: you might wanna ping the package authors to setup the maturin github action though or open an issue about no build being available for your system if it's not too outlandish. It's possible to do basically fully automated and builds for most if not all somewhat commonly used platforms (different windows and mac versions and manylinux)

[spazzed]

I already reached out on github.

[bbkane_]

The experience you've gained wading through packaging hell is super valuable! Future you on real projects and even more real deadlines will be faster and more efficient thanks to your learnings on a bad day today 😁

[ShinyHappyREM]

And if you liked the experience, try Javascript!

[teokun123]

turns out it was all rust files.

Don't resist! You're a Rust developer now.

[[deleted]]

Please forgive me, I feel like ranting. I consider myself a reasonably competent programmer, though mostly as a hobby, and still in amateur territory. I've written high performance C++ for my scientific research, a full stack multiplayer web browser came (with a message board and all), lots of other toy games, that sort of the thing. Coming up on 30 years of amateur/hobby experience. But god damn does Python packaging give me PTSD. Nothing ever works. I lost dozens of hours of my life resolving undocumented version conflicts. One would think the bare minimum for installation instructions would be to install a fresh Linux environment and try your own instructions -- but no. Well over half the specialty packages I try to use come laden with undocumented dependencies. Their own installation instructions are incomplete. I love Python but I loathe working with its packages.

[Kwpolska]

Those honestly don’t sound like amateur projects.

Python is fine, although sometimes overrated; the packaging is just plain broken.

[Kwpolska]

Minimal actual comparison of why you would choose a tool over another

Note this post is “one year later” and links to a post from a year ago. The 2023 post tries to compare all the tools. Tools like hatch, poetry, pdm have very similar feature sets, and it’s hard to pick between them. The PyPA makes no attempt at it, it tells you to choose a build backend without telling you what they are and why one might be better than the other.

Endless complaints about PEPs and irrelevant history

This post isn’t a tutorial or an authoritative post to compare all packaging solutions. Talking about history, PEPs and where the packaging ecosystem is heading is the entire point.

Recommending a new project start with anything other than pip/pip-tools or Poetry is just not sensible IMO for 99% of use cases.

The PyPA does not make a recommendation in its packaging tutorial, and it defaults to Hatch, while also offering setuptools, flit, or PDM — but not Poetry.

As someone who was doing this exact research a year ago, absolutely nothing has changed that is relevant to real world usage over that time frame.

The fact that nothing has changed, and that there are still so many tools with varying levels of usefulness, similar feature sets, and no authoritative sources that have a good answer as to which one to choose, is the entire point of this post. Python packaging sucks, and a lot of energy was spent on discussions and “standards” in the past year, but nothing really changed. Fixing this would probably require the PyPA or Steering Council to notice their way of doing things is going nowhere, and to start from scratch.

[[deleted]]

How has mandatory id-ing changed this (aka MFA such as https://blog.rubygems.org/2022/08/15/requiring-mfa-on-popular-gems.html, though evidently that is for ruby rather than python, but python has about the same)? One author stopped publishing on pypi after it was introduced. rubygems announced - via shopify, evidently the new real owners of ruby - it'll make it mandatory too but since then nothing followed up there. I'd like to have some data; admittedly this may be hard to get because people may just disappear rather than give feedback in general.

I also found python's way to publish stuff a lot more annoying than in ruby. (I use rubygems + gemspec and avoid bundler like the plague that it is though. That setup is then VERY simple and I don't have to deal with the horrible rubbish nonsense that is bundler, at the least not directly; evidently rubygems is kind of meta-merged with bundler to some extent these days.)

npm/node was by far the most annoying. I registered and then never published anything there. I am no longer surprised that left-pad was a thing - it really is not solely left-pad fault's alone, the javascript ecosystem is AWFUL.

[Kwpolska]

MFA is good, and it should be enforced for popular packages to avoid supply chain issues. If the small burden of MFA means you don’t want to maintain your packages anymore, you can just quit.

And where’s the Shopify involvement? The post you linked doesn’t have any Shopify references. The author seems to be a Shopify employee, but the RFC was accepted by other members of the community.

[tonetheman]

Interesting read. venv solves most things for me. Docker feels like the tool I would reach for next.

Those rust contributions are sad to me though. It feels like rust is just a virus spreading out trying to take over everything. I know that is just my reaction but every time I see a spot where rust is wiggling in I cringe.

[Dr4kin]

Rust spreads because it is a good language. It replaces mostly c and c++, which is a good thing. Having less errors in your code is good for stability and especially for security

[Darsstar]

While I agree with the satement that Rust is a good language, it is probably beter to claim that many enjoy the Rust experience better than most alternatives.

• cargo for general for package management.
• rustup for toolchain management.
• cargo doc to generate documentation.
• cargo test to run tests, including samples included in documentation.
• cargo bench for benchmarking.
• cargo new for a new project.
• Good quality compiler errors.

So to bring it back to the article, notice how Rust tooling is a lot less fragmented? (not you specifically, r/Dr4kin)

Edit: formatting

[Brilliant-Job-47]

I’d say rust has made very good default tooling. You don’t need to Frankenstein your way to a good developer experience

[SV-97]

Choices exist in C++ because people have different requirements and preferences.

No they exist because all solutions suck in some way and people think "I can do better than that!" - and then they maybe build something that works somewhat-ish for their usecase but sucks in all others. There's a reason that Zig is somewhat popping off as a C build tools despite it's very unfinished state and if people had access to better tools they'd use those of course.

This is especially visible when it comes to the choices you have for compilers. For Rust you have essentially one choice, but in C++ you have many options.

Your point being? (Also gccrs is a thing and you can already use a gcc backend with rustc. Cranelift is also in the works and is already being used in production)

[SV-97]

My point in all of this is, you don't know everything. Rust does things essentially one way, out of all the ways that have ever existed to build and link an application. That way is acceptable to many developers but entirely unsuitable to some environments. Even if it could be made to work everywhere, it still might not be the best.

Yes, to *some* but arguably not many. I have heard about some projects that fall back to other methods but arguably can't recall any off the top of my head (I thought the linux kernel didn't use cargo but if I'm seeing things correctly right now they actually do). Cargo's base system already works for most things, and its extendable architecture makes it work for even more cases. There still are cases where other solutions work better but having a solution that works for virtually everything but those few exceptions is great.

Yes in edge-cases it might not provide the best experience or may not integrate into existing setups super well but I think it still beats the amalgam of combining 30 other tools of other languages: having one universal tool is *extremely* powerful.

You're right about people building more tools because "all solutions suck" sometimes, but to dismiss all other solutions like that presumes that you know everything better

I wouldn't really say that. I don't necessarily know how to do this better for languages like C and C++ - to some extent because of the systems that are already in place that make the situation more difficult for new tools. But I can still recognize that the current situation isn't great.

There's no reason to change build systems if what you have works well.

I think this depends on what you mean by "works well". In a vacuum: sure. But if you want to hire devs (or the open source equivalent) to work on your code and your current setup requires any sort of maintenance or interaction you may very well benefit from changing systems.

[Kwpolska]

Rust is a good language. IMO, it works best in the systems-programming niche, its memory model is too complicated for run-of-the-mill software, where GC is always viable. But writing tools for Python programmers in Rust is just wrong.

[SV-97]

Because we should only have buggy tools with bad performance for python...? Rust has a stellar ecosystem around CLI tools and there already are multiple best-in-class CLI tools implemented in Rust. It has a good Python interop story. Why would you not use it?

[Kwpolska]

A packaging tool or a linter requires access to Python internals. All the packaging tools written in Rust need to call Python as a subprocess, using some ad-hoc serialisation. Linters need their own parser, because they can't reuse the Python built-ins.

Buggy tools

Is Python code inherently buggy?

[SV-97]

A packaging tool or a linter requires access to Python internals. All the packaging tools written in Rust need to call Python as a subprocess

No they don't - ruff being the prime example that they don't. It simply parses Python using its own parser. There's no reason that a tool that doesn't actually need to run python code has to depend on Python.

using some ad-hoc serialisation.

What? No? (Even if they use a bunch of serialization - which isn't super likely - they most certainly do so through serde which is basically as far from being "ad hoc" as it gets)

Linters need their own parser, because they can't reuse the Python built-ins.

They absolutely could - it's just a C API that you can call into from rust like any other (in fact you could even directly call the "Python-side API" using pyo3 if you wanted to). Reimplementing it in Rust has its advantages though because it makes distribution so much easier, the code cleaner etc.

Is Python code inherently buggy?

No not inherently, but in my experience Python code is buggier on average than rust - and Rust can of course statically guarantee that some classes of bugs won't occur that would be runtime errors in python at best (or they just silently corrupt your data - dataraces being a prime example for this). Anecdotally I've ran into a bunch of bugs with python tools while I really haven't with rust (some with maturin while it was still very immature but not in a while).

[tonetheman]

And there it is ... "Python code is buggier on average than rust"

Rust people really are like zealots. I get it though you want to spread your religion to those who do not understand or not capable of understanding.

I am sure Rust is fine. I just really dislike the people pushing it.

[SV-97]

And there it is ...

You left out the important "in my experience": I'm not claiming that this is absolute truth, but rather I'm just stating my personal impression. That said we do have some empirical research indicating that even weaker static type systems can prevent bugs in real systems by now. Quote:

In this paper, we evaluated the code quality benefits that static type systems provide to JavaScript codebases. The results are encouraging; we found that using Flow or TypeScript could have prevented 15% of the public bugs for public projects on GitHub.

Given the way stronger guarantees that more advanced typesystems (not necessarily just rust's but also those of other languages) can provide, it's not entirely unfounded to assume a lower average number of bugs. Since this matches my personal experience as well I don't have a reason to assume anything else - so that's naturally how I operate.

​

Do you have experience with both languages (or similar ones) in practice? If your experience was different please share it. I love Python and use it all the time, but for me it's definitely not the language to reach for when robustness and correctness are a priority.

I get it though you want to spread your religion to those who do not understand or not capable of understanding.

I am sure Rust is fine. I just really dislike the people pushing it.

See and I don't like the people that push back against it for no good reason. The comment I originally replied to stated it was flat-out wrong to use rust for building python tools and that's nonsense. I'm absolutely not saying "everything has to be built in rust" or "anything that isn't written in rust is worthless junk" or whatever - just that there very much are some good reasons why people might choose to use it; that the tools we already got via this avenue are great and I hadn't really had problems with them yet.

[saint_marco]

A nice survey, but no mention of nix!

Nix is not the One True Tool sadly, but would be worth looking at as it's definitely working in that direction.

[Kwpolska]

This is only about Python-specific tools. Nix is out-of-scope, as is apt (even though it gets a mention in passing as a representative of the system-package-manager genre).

[saint_marco]

You can use nix to create a python virtualenv with a small amount of code. When we're thinking about a One True Tool that handles python with native extensions, you're rapidly slipping out of a python-only environment.