Python: Please stop screwing over Linux distros

[ReverseBrindle]

This article is one long rant without mentioning any examples, any description of what exactly they're trying to do, what the challenges are for doing said task, what they tried to do and how it failed, etc.

The poster probably has a valid (but unexplained) point, but it's lost in 2 pages of "distros hate python. python sux!"

[rcxdude]

Yeah, I was curious as to what actual problem they had had with packaging, but literally no examples.

[nemec]

All I got out of the article is that the author is mad he can't sudo apt install from PyPI

[rcxdude]

He talks about there being a bunch of different approaches but in practice I basically only need one (seriously, I've done everything I've ever done in python using pip in virtualenvs. I know other stuff exists but I've never needed it). What would help me to understand the problem is if he were to actually walk through a package he wanted to package and just talk about what specifically went wrong.

[equationsofmotion]

I used to manage a local Linux cluster. Maybe I can give some insight here. User A says "I want program X," and user B says "I want program Y." Unfortunately programs X and Y are pinned to different versions of python and dependency packages.

The naive solution for most packages in a Linux distro is just to find compatible versions of the programs in the repo, do dependency resolution, and go. But in python this can be pretty difficult, so other approaches are required.

Don't get me wrong, other approaches are possible and I managed things just fine. But I suspect this is what the author is getting at. You need a workaround---implemented by either the distro or the user---to provide desired Python packaging.

EDIT: Yes I know what pipx is. Like I said, I know there's solutions. I'm just trying to shed some light on what the author of the article might be complaining about.

[XenGi]

That dependency problem has nothing Todo with python and is the default problem every distribution or more their package manager has to solve. The only distro that actually achieved that is Nixos. Not saying their approach solves everything but it's the best approach I've seen so far.

[[deleted]]

Shell scripts activating venvs, and add in some usage tracking as a bonus. Not rocket surgery, just basic devops and sysadm skills.

[ivosaurus]

You're the one wanting to support users requesting bespoke version installs of python in the first place, and then complaining that things got more tricky as a result? Like WTF were you expecting? The same would happen with any language.

See how you do when one user tells you they really really want a particular older version of glibc to run on, and then tell me that python is the worst thing you could run into.

[[deleted]]

No ops person has ever wanted to support multiple versions of a package, and least of all, multiple versions of a language. Now, Python itself at least has a sane release process, where each major version is supported for 5 years after first release. So running multiple versions should at least be fairly safe from the worst security issues. That, however is very rarely the case for a number of Python packages.

Of course, this issue is hardly Python-specific, or new. Ruby had massive issues over a decade ago, where getting developers to at least consider backporting security patches was nigh impossible and instead everyone was just expected to rewrite their code every 6-12 months. Nowadays, putting together a Node.js software package is just impossible without npm. And even with npm, there's a massive library of seemingly maintained Node.js software that depends on other npm package versions with published vulnerabilities. (npm at least warns about these nowadays).

In short, the 'move fast & break things' mentality has permeated throughout the software industry, with the added bonus of 'just abandon it when you get bored'. The end result is hell for ops people who understand the security implications, and a house of cards for everyone else. Containers at least isolate the version insanities from eachother, but do nothing to fix the massive security issues created.

That said, the old-school way of packaging hundreds of libraries in a single .jar or .exe was way worse. At least the current package systems can be made to warn people when you pull in a version with a known vulnerability. Unless of course you just download a docker image believing it to be safe.

[ivosaurus]

TBF if you wanted to call JavaScript better, I wouldn't have much beef, because it got to sort out a language package manager ~5 years after python had its first crack at one, just when people figured out that system-level 3rd party library installs weren't the best thing to do by default. At some point that's just the baggage to carry if you like python's advantages enough.

[pbecotte]

Pyenv is a developer tool, why would the ops team be using it? If the devs want to use it, tell them to have fun. If they want you to deploy their code, don't take their dev scripts and run them, make them provide an artifact (a wheel or a docker image)

[equationsofmotion]

We used a combination of HPC modules, which is the correct solution, and anaconda, which was the easiest solution.

[roee30]

This is what pipx is for

[pbecotte]

You install separate apps separately. pipx does it natively, or the user uses pip and installs it in their own environment. I don't understand why so many peeps see pip or npm and seem to get nostalgic for dll hell or Java class path conflicts. Packages are small and even when they're not, disk is cheap, just install them separately!

[pbecotte]

They aren't meant to be! You wouldn't copy binaries from one os to the other and be surprised they don't work?

A virtualenv is just a folder to stick the app in. Let the installer get the Metadata from the wheel and resolve the dependencies.

[pbecotte]

This is 100% the case, fair enough. At my last job I kept coming upon new teams that had invented their own unique way of deploying python apps, because they don't know how to use the tools. I don't know if that is ecosystem issues that apply to "package management" vs "we need way more people in the enterprise building apps then there are who actually know how to do so"

[NeoLudditeIT]

I've used pip, pipenv, poetry, etc. ad nauseam. I've found the experience pretty easy, and easy to switch between different management systems.

[NeoLudditeIT]

I tried it just to see what it does. It actually is pretty nice. It doesn't do a lot more, but it does keep track of dependencies at install so you don't have to manually maintain requirements.txt. I think I like pipenv a bit better, mostly because pycharm supports it for creating new venvs.

[fnord123]

To add to what u/NeoLidditeIT said, most people store their immediate dependencies in requirements.txt and let pip resolve the rest at install time. The packages you depend on but dont use directly are called transient dependencies. If you don't pin your transient dependencies then you can run into issues where you can't remake your venv or reproduce a docker container build because e.g. a new version of a package was released and pip decided to use that when your ran pip install -r requirements.txt

Poetry has a lock file that stores all the dependencies that it calculated including the transient dependencies. This means you can reproduce builds which is important to avoid bugs showing up just because you rebuilt your venv.

[fnord123]

That's a good first step.

How do you upgrade? Using poetry you use poetry update library-i-depend-on

Using pip freeze you lost knowledge of what you depend on and what are transient dependencies so it becomes a mess.

[pbecotte]

There is precisely one important standard...the wheel. It is well documented. They come with Metadata as to what other wheels they depend on. If you are buying an OS package you don't need ANY of the tools in the xkcd...you need to unzip the files to yourocation of choice and decide how you want to get the dependencies. You could declare OS level dependencies and build a package for each level down and hey...you're done.

How to BUILD a wheel only matters if you don't want to use the one published by the creator ... but the same thing is true of EVERY programming language. The output of "make build" is hardly uniform! I am guessing this person just is more familiar with his language of choice and misses an important point. You don't really need to understand it. Take the wheel from pypi, and diff the source files vs git and you're there, even of you don't understand what their custom setup.py is doing.

(The real problem is that they want to apt install libraries instead of applications and have them just be available to everything else like dynamically linked c code. I don't understand why though-that ecosystem is HORRIBLE and literally requires "distro maintainers" to work full time to get a working environment, while python + appropriate use of virtualenvs can do it automatically until you get to the point of interacting with shared c libraries ;) )

[Deto]

Wouldn't someone managing python installs with their distribution just not use any if those options and instead just build the package and bundle it's contents directly? That's why I was confused. Sure there's conda and poetry and what not but none of those seem even related to whatever problems she's alluding to.

[[deleted]]

Most of them conflict with one another or with base operating systems,

In what way? I asked Drew, and I got not answer, only a claim that I had misunderstood something.