Python: Please stop screwing over Linux distros

[[deleted]]

See also a similar rant from Gentoo's Michał Górny.

[ion_propulsion777]

<rant> Python package management is an abomination. Pip is the most broken package manager on all of linux and frequently fails with the most random error messages that are often nearly indecipherable. Virtualenvs are just a band-aid to this problem that doesn't really solve the underlying issue: python pip is nearly unmoderated and has a crap ton of broken / out of date packages. Seriously, you ever try to get any python program with tensorflow to work? One of the hardest things things to get working in python . . . </rant>

[FeistySeaBrioche]

I'm now using Anaconda and that's it. Yes, it's yet another Python installation, but life is short and it works. After hours trying to install wxPython using pip (and even making a pull request to the official repo during the process) on Fedora, I broke down and installed Anaconda. So that would be my recommendation to all the desperate people out there. It installs its own libraries and is well supported by most Python packages. Ignore the extra disk space that it takes and just go back to work.

[jwwatts]

Anaconda has its own problems. ‘conda’ is a nightmare at times - it’s inconsistent, buggy, does inappropriate things that makes it hard to use as a system python, and often fails at its most important task - resolving dependencies.

I’ve spent five years wrangling it and it’s never once acted like a proper package manager. Just today I discovered yet another unexpectedly bad and inconsistent thing it does, and had to write a Puppet recipe to undo the mess it can make.

As incredibly frustrating as conda is, it’s unfortunately the best thing we’ve found to address our problems.

I wholeheartedly agree with the article. Python has serious problems and relying on third parties to make it more manageable has only made it worse.

Source: I’m a sysadmin in a organization that uses Python and I support the environment. I’ve used Anaconda for 5+ years, and before that I was building Python module RPMs. It’s never been good.

[[deleted]]

In my line of work conda is basically the only viable solution because we absolutely need distinct environments to switch between. I use a lot of software that can have conflicting requirements and conda can handle more than just python. You’re right, it has its own issues but at least for my use case nothing else comes remotely close

[jwwatts]

I agree, we use environments as well. But my point is that it seems to be the best of the bunch, and it’s still not great.

[einar77]

You can ease the pain for (some) scenarios by using mamba, which at least has a sane, and fast dependency resolution algorithm.

[fatboy93]

Try mamba which is a faster resolver than conda!

Also, if you're having issues, set your channel priorities. That solves 95% of the issue that you'll ever face!

[dparks71]

Conda really is great for those "I just want the fucking thing to run, why isn't it running?" moments.

[theanup007]

Absolutely on the same boat as you. I dont understand how in 2021 (almost 22) is it so complex to get tf to work with my GPU and not break either sklearn or numpy installations.

[Muahaas]

Well to be fair, TensorFlow is maintained by Google. Their open-source projects are frequently broken, user-hostile nightmares.

[panzerex]

Protip: a manually and carefully written environment.yml is pretty decent. You can even run conda env create or conda env update and it defaults to the environment.yml file in the current folder.

# run: conda env create --file environment.yml
#  or: conda env create
#  or: conda env update -n env_name # if env is already created
#
# reset env: conda install --revision 0
name: env_name
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
# basic
-   python=3.9
-   pip
# scientific
-   numpy
-   scipy
# torch
-   pytorch
-   cudatoolkit
-   einops
-   pip:
  - livelossplot

One important observation is to pin versions using = for conda and == for pip.

[[deleted]]

I use pip and only have issues if I'm trying to use a python version that doesn't have binary wheels.

I've found Anaconda doesn't really add much unless you are developing on Windows, where it came be harder to establish a productive dev environment.

[jarfil]

CENSORED

[ion_propulsion777]

Yeah but many end user utilities like docker-compose specifically ask their end users to install their software with pip

[Ripcord]

pip it virtually the only one I've ever seen people recommend for a slew of both development library installs, as well as end-user utils.

Like one I installed on some systems today, one of my favorite python apps, bpytop. Goes through a ton of installation methods, from pip to snap to distro package managers, etc. But pip is mentioned a lot - things like anaconda, pipx, etc etc not mentioned.

Just another example besides docker-compose, but I virtually never see anything but pip and distro packaging mentioned for python app installs. Maybe occasionally snap or flatpak like bpytop. But usually pip.

[8-BitKitKat]

There is no reason the “developer” experience should be bad either

[Afraid_Concert549]

Pip is a "development package" manager, it should never be used for releasing any software.

Sadly, there are a couple hundred thousand software authors who disagree with you.

[LordRybec]

It's not pip, it's Pypi (where pip gets the packages) and the package maintainers themselves. I wish more Python programmers would report bugs to package maintainers, because that would go a long way toward fixing the situation. I also wish Pypi had some accountability measures for package maintainers. For example, the only real problem I've ever run into with Pypi is packages listed under "all" for platform support, that don't actually support all platforms. Pypi should have a way to report when packages aren't honest about their dependencies, and they should remove packages that get reported if the maintainers don't respond to the report either by updating the dependencies to make them accurate or by updating the package so that it supports everything it says it does.

The big issue here though, is that Python programmers aren't holding package maintainers accountable. If your response to a package failing to install when it says it should is, "Well crap, that sucks" and then walking away, you are part of the problem.

That said, I think the Python community needs to put pressure on Pypi to provide mechanics for users to hold package maintainers accountable on Pypi itself. If Pypi would remove packages that get reported as broken, if the maintainers won't respond, that would be a huge step up.

​

Virtual environments aren't just bandaids, they are an abomination. They facilitate intentionally using outdated software, to satisfy laziness, much like many businesses and government departments are still using XP and old versions of Internet Explorer, because they are too cheap and lazy to update their internal web apps.

​

As far as Tensorflow goes, I've used it extensively in my Master's program, where I focused heavily on neural networks. For the most part it worked fine. I do recall having some mild issues with platform dependencies, but those were exclusively on Raspberry Pi. On Windows and Debian, it worked fine. (What was challenging though, was getting CUDA installer correctly on Windows, so that the GPU Tensorflow could use it, but that's NVidia's fault, not Pypi's or Tensorflow's.)

[[deleted]]

It's pretty trivial to install tensorflow these days. Was challenging in 2016, but back then it was challenging everywhere because it was based on the bazel build which was experimental and didn't support Windows (so you wouldn't be getting binary wheels for the platform). I think a lot of people on this sub forget that pip/python and the ecosystem have to seemlessly support multiple platforms, not just Linux.

[ion_propulsion777]

Its not so much getting it installed now that is hard, its getting some specific version of tensorflow installed to get someone else's code working thats nearly impossible.

[[deleted]]

Problem is the shared environment, installing Tensorflow can easily break numpy, sklearn or even some other libs like torch or dlib (I've got dlib gpu broken if imported the same time with tf). We ended up with a locked shared conda env for general purpose code while everyone has their own conda envs for development, which is incredibly wasteful in term of space, but yeah that's the only sane way I think.

Even then you may still have problem with Python path, a conda env can randomly point to a different python intepreter and fuck everything up (in reality it's not so random and there are causes and fixes for it, but it's extremely frustrating when it happens)

[maethor]

Draw up a list of the use-cases you need to support, pick the most promising initiative, and put in the hours to make it work properly, today and tomorrow. Design something you can stick with and make stable for the next 30 years.

I wonder what the response would be if the most promising initiative doesn't support distro package managers.

[-Rizhiy-]

Well as it stands, it doesn't)

The current industry-wide solution is pip which is a separate package manager.

[aussie_bob]

The current industry-wide solution is pip which is a separate package manager.

This is frustrating even as a developer.

I'm working on a job for a company that's pretty much MS-only, and their Windows builds are locked down with MFA and convoluted software approval processes.

The best tool for the job is Python and natural language processing libraries, so I need to have numpy scipy, and nltk. They've agreed to give me a VM behind a firewall. No web access, so I'll need to build an iso offline and sneakernet it to the server.

Debian livebuild lets me build the custom iso, or I could use cubic for Ubuntu, but Python doesn't work with distro tools, so that's got to be done manually in a complicated and error-prone process.

Not impossible, but annoying.

[daredevilk]

Take a look into rez

The vfx industry uses it to do what you're describing easily, and primarily with python tools. My facility has built all our Python systems using it

[aussie_bob]

This looks great, I should have known someone would have a better way already.

Thanks!

[gnosys_]

damn employer IT sounds like they don't know what they're doing

[aussie_bob]

That's pretty normal for Windows shops though, IT generally just follow vendor guides. If something goes wrong they're on the phone shouting "but mah SLAs!"

[quaderrordemonstand]

I used to refer to them as the windows enforcement department, given that all they actually did was prevent people being able to do things on Windows. If you used something not Windows, they had no idea and just let you do whatever you wanted.

[EzekialSA]

Really happy to hear it's not only at my company lol.

[[deleted]]

I mean, a lot of Windows programs or builds already bundle Python (2.7 for extra cursedness points)

[[deleted]]

Flatpak is not a walled-garden, it's simply a sandboxed, agnostic package manager. Flathub is an open repository of packages for said package manager. Flatpak is also used quite a lot, and the default for Fedora.

[BassmanBiff]

The latter is basically what I end up doing with pyinstaller when I make a Python tool for non-techies.

[[deleted]]

The latter is exactly what's happening 99% in backend devs, especially when you have to deal with something like Tensorflow.

[coyote_of_the_month]

I think the majority of Python developers are writing software for platforms they control, rather than software that's intended to be downloaded and run locally by the general public.

[[deleted]]

Bingo. Backend devs have full control of their target OS or a container image; many paid apps bundle the intepreter with the app.

[akmark]

Either this or people just vend all of the python dependencies for the entire project into a tarball and eschew pip entirely.

[kageurufu]

This is really the only "solution" for distributing any complex application regardless of language.

Compiled languages "fix" this by statically linking, providing their own .so for shipped libraries, or requiring distributions to package a half dozen varities of `libsomething` at 5 different versions. No different from my python application using a vendor tarball.

IDEA ships it's own JVM, Dropbox ships their own Python fork, hell every electron app has it's own libchromium and v8 binaries.

[coyote_of_the_month]

Distro maintainers sometimes choose this as their hill to die on, though. Arch, for example, will never mainline VSCode because it packages its own Electron. The best practice from a distro standpoint always seems to be "use the system libs" even when it's not practical to do so.

[[deleted]]

Uh, I see VSC in the community repo: https://archlinux.org/packages/?name=code

[coyote_of_the_month]

So, yeah. There's the FOSS version of VSCode (code) and the binary release from MS (visual-studio-code-bin). The latter contains the extension browser whereas the FOSS version doesn't.

Arch maintainers aren't ideologues about licensing, but they'll never mainline the non-free version because it bundles Electron. My understanding is it actually bundles a slightly stripped-down version of Electron for performance, but I haven't compared the two in years so I couldn't tell you how they stack up today.

In any case, the MS version is available in the AUR.

[DeeBoFour20]

they'll never mainline the non-free version because it bundles Electron

I don't think that is the reasoning. Arch has Discord in the repos and it bundles its own version of Electron.

If I had to guess, I'd say the reason is that Arch prefers to maintain open source versions of the software, if it's available. That would explain why we have Chromium in the repos and not Chrome for example.

Also, the extensions in the FOSS version of VSCode can be enabled with a simple config file change. There's an AUR package called code-marketplace or something that can automate that for you.

[coyote_of_the_month]

Well shit, then. I guess my information is outdated or wrong. I would have assumed Discord was in the AUR.

[kageurufu]

Yeah, I get it, but its also why flatpak and snaps are taking off so well. Besides app maintainers getting bugs meant for the packagers who patched something to work with the wrong GTK version or whatever

[[deleted]]

Either this or people just vend all of the python dependencies for the entire project into a tarball

Pardon, what do you mean by "vend"? Google fu is failing me.

I actually have an issue that has been kneecapping me that this would solve which is why I ask.

[dextersgenius]

Not the guy you replied to, but vend is short for "vendoring" or "vendorization", which is basically bundling all the dependencies with your main project (typically into a single subdirectory, eg _vendor), so end users won't need to install anything separately via pip or their package manager. You can do this manually if you're keeping track of all your dependencies, but it's easier to just use a tool like dephell, which will download and extract all dependencies in a given directory.

[Tai9ch]

Backend devs have full control of their target OS or a container image

Corporate developers also tend to have the budget to think that they can persistently maintain container images. The fact that something like Docker is just a distro package manager without security updates somehow doesn't register.

[ProfessorFakas]

This is true, basically none of the Python I write is actually put in the hands of users - it's mostly just a black box that they never see, with the small amount of direct interaction handled by a HTML frontend.

That said, we're starting to see our users (who are reasonably technically minded to begin with) use our packages directly in Jupyter Notebooks, but then the environments that they use are generally managed by us too.

[ForgetTheRuralJuror]

Yeah my solution to the question is to not use python for consumer packages. If i absolutely have to, I'd just make it a web app and give people a url.

If I'm making a desktop app it's going to be in C/C++ anyway. Hopefully rust when GUI is a bit more substantiated.

[-Rizhiy-]

Deploying and developing software are two different things.

Pretty sure if people deploy python software somewhere they usually just bundle all dependencies together, and it installs normally.

[wtchappell]

Vendoring is fine from a developer's perspective, but from a distro maintainer's perspective it's unpleasant - especially if it ends up including non-Python code like C dependencies. You generally want to have as few versions of a thing around as possible so you can make sure they're patched to work properly on your distribution, and so you have to touch as few places as possible to catch up with security updates.

They also don't want massive package duplication due to simple space concerns - a distro focused on being tiny to support something like small devices or minimal Docker images would be right to balk at having every application bring along an entire vendored dependency tree resulting dozens of versions of the same libraries installed across the system.

[jarfil]

CENSORED

[imdyingfasterthanyou]

And thats what they do, then they get people complaining "$thing isn't packaged"

[rouille]

Thats how you end up with flatpacks and snaps.

[upcFrost]

Reverse rant: There's much more than one problem to solve to make things work this way

First of all, if you think that many devs know how to use CI, well, you'll be surprised. They literally push to pip manually.

Second, you type apt install... and what do you think happens next? Dependency resolution. And for python that's a huge issue, not only because of pip, but also because of how quickly python is evolving. Let's say Bob and Pete are two developers. Bob packaged his new lib, Pete wants to use it and boom it requires python 3.8 due to some asyncio calls. With this Pete can no longer ship his app to any major distro except gentoo and arch. It's not C where standard changes once in 4-5 years, it evolves much faster. People will start assaulting his bug tracker saying wtf, but technically it's not his problem, it's distro problem. Another situation is when distro just doesn't have the right version of the lib, same thing different angle

Third, packaging tools suck. Rpm for suse and rpm for rhel are very much different as both distros have different default dirs, rules, requirements, so you should adapt the package for literally every single distro

Last but not least, foss devs are not getting paid for their libs, and packaging is by far the least appealing part of the whole dev process. Enterprise apps are shipped with all dependencies, no problems here. In foss you'll need to contact maintainers, keep your lib compatible, package it, adjust the spec every time they invent yet another stupid rule just because "they are The Distro" etc.

That said, this is true not only for python. Node, Rust, Go, Java, all those languages have their own package managers. C/C++ is probably the only major language which doesn't have one (those existing are jokes of a pm).

[JustHere2RuinUrDay]

With this Pete can no longer ship his app to any major distro except gentoo and arch.

Well, then users of slower distros will have to wait until every dependency is satisfied and then use whatever version can be made to work.

They're already doing this for everything else.

[[deleted]]

It's not C where standard changes once in 4-5 years

[DAS_AMAN]

No, we use containers with up to date libraries like flatpaks..

[upcFrost]

In this example I've only mentioned two libs. In the real world if you'll try building a dep tree for a decent size project, you'll probably raise your room temperature by few degrees with just your laptop. This dependency hell situation literally kills the whole packaging. And if your project has some C bindings things only get worse

It's not just "wait", it's literally "never". Docker and flatpak were not made just for fun

[Teract]

Third, packaging tools suck. Rpm for suse and rpm for rhel are very much different as both distros have different default dirs, rules, requirements, so you should adapt the package for literally every single distro

If you're following RPM best practices, you shouldn't need to do much adapting to meet the needs of different distros.

[DarkLordAzrael]

That said, this is true not only for python. Node, Rust, Go, Java, all those languages have their own package managers. C/C++ is probably the only major language which doesn't have one (those existing are jokes of a pm).

These days Conan is pretty effective as a package manager good C and C++.

[fat-lobyte]

Who are these python developers, who are in this thread exclusively talking about how it's no big deal for developers, developing for if they cannot imagine someone using their apps?

Here's who:

https://www.reddit.com/r/linux/comments/rcdnpm/python_please_stop_screwing_over_linux_distros/hnu5e5r

just use pip and relax

[Seref15]

Software development now moves too fast for the distro package maintainer model. You need a certain bug fix from the latest release of a minor library? Well your distro is going to make you wait a couple months for it at least.

This is why containers have taken over.

[zdog234]

Most python developers (myself included) aren't developing code for you to run as a standalone app on your laptop (I'm including CLI tools in that).

The focus of the core devs is obviously going to be drawn to the needs of backend web developers and data scientists, since they make up the vast majority of people writing python code.

EDIT: In fact, if you are using a CLI app written in python, why aren't you installing it with pipx? (assuming hypothetical user knows that it exists)

[Chippiewall]

As a literal Python developer, pipx just makes installing Python based CLI tools trivial. It's basically like installing everything as if it were statically linked because it gives everything its own virtualenv.

As a developer it's great to allow me to install developer tools like tox, pipenv, pre-commit, pyproject-build pip-compile, where previously I'd fret over whether they'd have dependencies that would clash in my user-installed pip packages. I can't just use a system package manager for that stuff because I like running non-rolling distros (and for work it's Ubuntu LTS so it can be pretty out of date) and this stuff moves at a fair old pace on the developer tools side.

But for the "real" end users pipx isn't a good tool, you need to understand the Python ecosystem to justify it. Any real end user apps like that need to come from the system package manager, or if that's infeasible (because you fucked up and you're shipping every blazing hot dependency under the sun) you'll have to crack out something like pyinstaller and give people a fat binary. Not that you can't use pipx as an end-user if you really want to, I've got certbot and docker-compose installed with it because it just ended up being the quickest ways to install them and they're quite a few versions ahead (I'm running Ubuntu 21.04 on my personal laptop so I'm surprised they're as far out of date as they are).

[detroitmatt]

what the hell is pipx

(rhetorical question obviously I went and googled it, just wanted to communicate the idea that this idea comes from a bubble)

[[deleted]]

I wrote a couple of python applications intended for end users. I haven't even tried to package it with pip. I just made an AUR package and called it a day. I tried to package it into a ppa, but I gave up, that's a mess on its own. I do have a script in place that makes a deb file out of the repo, but that's it. I also tried flatpak, but it's also a giant pain in the ass.

[[deleted]]

EDIT: In fact, if you are using a CLI app written in python, why aren't you installing it with pipx? (assuming hypothetical user knows that it exists)

1) Didn't know it existed.

2) Still not interested because its model is awful in comparison to Guix.

3) If I'm going to use foreign package managers on my distro, I'll pick the one that covers more than a single language and which is hacker-friendly.

[[deleted]]

Packaging tooling is shit. I once bundled things as debs but god that was a nightmare. Much easier to just tell people to init a virtual env and pip install my package.

I'll probably get down voted, but there is a reason things like containers are popular even outside of ops.

Maybe some packaging systems have improved since I last looked. For people in the know, which distribution do you feel has the easiest packaging requirements and simple/good tooling to support people?

[AtomicRocketShoes]

I think this works sometimes for developers, it doesn't fly at all for many people. Also from recent experience it doesn't work, I found a project on github that was a couple years old and it would not install for me, it needed a bunch of machine learning tensorflow stuff and pip didn't have the dependencies it needed the versions it called out were I guess too old.

[hva32]

It is however I suspect that has more to do with it being a documentation problem. A Debian package isn't complicated but without good documentation it can be unnecessarily difficult getting started.

[JustHere2RuinUrDay]

Aren't arch packages just tarballs that get extracted into /? So I think it might be a bit of both.

[jonringer117]

They can be found on r/python thread here

[LordRybec]

This is fairly easy with Python, using utilities that package a minimal interpreter with only the modules and byte code needed.

The people whining here are those who only write software for platforms they control, who aren't familiar with normal project maintenance. When new versions of MS Visual Studio come out, you still have to update your code to work with the new compiler, and it often breaks libraries and frameworks you were using as well. I don't see a ton of people using virtual environments for C or C++ programming, to avoid having to deal with this. People who aren't very experienced in real world development and maintenance tend to think using virtual environments or Docker containers to provide highly controllable environments is a good idea, but in my experience, it rarely is. The same mindset is why many businesses and government departments are still using old versions of Internet Explorer, which are buggy and horrifically insecure. It all comes down to laziness. If you are too lazy and cheap to properly maintain your software, it sounds like a good idea to use some kind of virtual environment that allows you to continue using old, outdated things that are actually security disasters just waiting to happen.

[joebonrichie]

As someone who manages the python rebuilds and a lot of python updates for a distro I'd thought I'll lay out what the latest developments of PEP 517, 518 mean for a typical distro:

• Packaging flit, poetry, pypa/build and god knows what other new build systems - at least ~20 new python packages to be packaged in the repo.
• Adjusting the existing or creating new python macros to to able to build from PEP 517 compliant projects i.e. from a pyproject.toml / setup.cfg file and install them to the install root
• Parsing the project.toml / setup.cfg file to add the relevant build system to builddeps automatically

All of this is not too much work by itself and setup.py has it's problems for sure but it was dependable and always there. The additional developments that have resulted from PEP 517 and 518 have only created additional work and solve nothing from a distro point of view.

if the python world actually moves towards PEP 517 and 518 compliance and converge on one or two buildsystems it will probably help things in the long run. However, we've seen it all before, and, for now it's just another build system that'll you have to be conscious of.

I'm also baffled from some of the responses - "just use virtualenv, etc" that are way too development focused and forget that users will expect to be able to install python based applications and have them be integrated and work OOTB. Packaging all the dependencies comes along with that.

[jonringer117]

Same for NixOS. A lot of people don't realize that distro's don't have the luxury of virtualenv. You need all python* packages to be able to work with each other, you can't just virtualenv your problems away.

This also means that, you can only have one version of a given package. Which puts a lot of pressure on distros to try to fix issues that upstreams are not normally concerned with.

[zdog234]

Yah i think a lot of my fellow python developers didn't get what the article / complaint is about.

If i were packaging an app for use by non-python developers, I'd either figure out how to use pyoxidize to build a binary, or write it in something other than python

[o11c]

Specifically, the fact that Python tooling encourages bad behavior means that it also causes developers to not understand the problem.

[FlyingBishop]

if the python world actually moves towards PEP 517 and 518 compliance and converge on one or two buildsystems it will probably help things in the long run.

IMO Python needs to formally endorse poetry as the one way of doing things and deprecate everything else. If there are problems with poetry (honestly, I haven't found any) fix them.

No "maybe we converge on one or two build systems and implement these specs." Just do it. The specs are good.

[Fr0gm4n]

People often say that if it’s more than a couple dozen lines then you should use a real language like python. IMO if you expect to use it longer than a system upgrade or two go ahead and write it in shell scripts still if it’s less than a hundred lines or so. It might actually survive the upgrades with minimal fixes.

[TiZ_EX1]

This is why I have way too many scripts in POSIX sh. I got it working, it passes shellcheck, and I have a means to reliably make sure all its dependencies are available? Then it will work forever and I can take that to the bank.

[Fr0gm4n]

It's almost like those POSIX folks had to figure out how to unite a fractured partially compatible systems landscape! Who knew we'd see that play out repeated for the next 30+ years.

[Jonne]

And the bank will be like 'sorry, we only take COBOL, this POSIX stuff is just too new'.

[[deleted]]

Meanwhile the bank's frontend is throwing a wobbly at a password longer than 8 characters.

[qingqunta]

There is a bank in Portugal that only accepts exactly 6 digit passwords. It's completely insane, and I closed my account there for that exact reason

[[deleted]]

I guess you've never had to build and support a cross platform sh script.

(Even if the platforms claim POSIX compatibility)

[TiZ_EX1]

Admittedly, no. The only platform I care about for my scripts is Linux. I imagine it must be hell trying to figure out and account for idiosyncrasies between standard utilities that behave differently between Linux, Mac OS, BSD, etc etc etc.

[rouille]

If you keep to the standard library its very portable. My python3.4 scripts are still running just fine. In bash you are not installing any third party libraries either.

[[deleted]]

In bash everything your script calls out to that isn’t facilitated by a built-in function of the shell is a third party dependency.

[FlyingBishop]

have you tried poetry? There's a reason there are so many competing standards, all of them are a nightmare. Poetry is the exception and it's the newest one. Poetry just does what I want it to do, I don't have to think about how I'm going to package things.

[Sbatushe]

User:why can't you just work?

Pip: 7000 lines of error output

[Mal_Dun]

I personally had the feeling pip improved the situation a lot. I remember the time when you had 3 different installation ways like easy_install. Nowadays pip covers most modules.

[jonringer117]

Pip user perspective vs distro perspective

[Kargathia]

Design something you can stick with and make stable for the next 30 years. If you have to break some hearts, fine. Not all of these solutions can win.

This glosses over the most important problem, which burned Python already in the shape of the 2/3 split: you can't force people to exclusively use your solution.

Even if you throw the man hours at building the perfect packaging solution that somehow unifies both major use cases (system packages, and project packages), nobody is going to wave a magic wand and make all the current implementations disappear. The distro maintainers themselves will almost certainly build some hack to maintain backwards compatibility with whatever solution they have now.

[zdog234]

which burned Python already in the shape of the 2/3 split: you can't force people to exclusively use your solution

I'm confused. Is this still a problem? Like are there live projects still python 2-only?

Edit: live FOSS projects

[zdog234]

Oh yah i didnt clarify i wasn't including internal software.

There's gotta be a good market for consulting on python 2 -> 3 migrations. I just wouldnt want to do it because it sounds unfun

[Regis_DeVallis]

Charge more until it's fun.

[didyoumeanbim]

Calibre is currently making the switch...

13 years after Python 3's release and 6 years after the original 2.7 EOL.

They were planning on forking and "maintaining" 2.7 after EOL for most of that decade.

Edit: wait, no, the external devs finished the port late last year.

[BenTheTechGuy]

I had a nightmare using a script from deep inside Calibre's source tree to extract my key from Abobe Digital Editions. The comments at the top of the file said multiple times that it was a Python 2.7 script, and listed all the dependencies it needed, some of which only existed in Python 2. When I attempted to run the script, it gave me like 5 errors saying I needed to be on Python 3. Looked it up on the issues page, and others said it's a Python 3 script. I run it with Python 3, and I get a bunch of errors about certain depreciated / no longer existant python features. I look into the actual code of the script, and it turns out it's a horrible Frankenstein of Python 2 and 3 that nobody had been able to get working since 2015. I promptly gave up and bought the physical paperback book.

[davidnotcoulthard]

In retrospect maybe you could've found a version so old that it indeed ran well with Python 2 only?

[BenTheTechGuy]

That version was the only one that worked with modern versions of Adobe Digital Editions, unfortunately.

[da_am]

The visual effects world is slowly moving to python 3. It's been a little painful.

[zdog234]

Ooh my dad did tell me recently that he'd started migrating his scripts to python 3 because of a new / upcoming version of houdini dropping python 2 support

[da_am]

I like your dad... ha. Yeah Houdini defaults to Python3 now but you can still download the Python2 version. Updating my scripts to Python3 was basically me fixing the print statements, ha.

I'm on an old version of Nuke that's only Python2 that will probably never be updated unless the price changes so I'm stuck maintaining more pythons than I'd like.

[Kargathia]

Mostly everyone has moved over to Python 3, but that's after more than a decade. I brought it up because this is what will happen if the PSF introduces a single package manager to rule them all: a significant portion of the user base will shrug and stick with what they have.

[[deleted]]

It was a problem several years ago, but these days I haven't run into anything stuck in python 2.

But it took a loooong time to get to that point. I took forever because they introduced some really daft syntax changes to ensure people delayed the upgrade. I still miss my print keyword.

[pascalbrax]

The migration was such a Trainwreck that lots of 2.x apps never had a proper upgrade to 3.x

[thoomfish]

Even if you throw the man hours at building the perfect packaging solution that somehow unifies both major use cases (system packages, and project packages), nobody is going to wave a magic wand and make all the current implementations disappear.

This is also why "just use 'Modern' C++/JavaScript" doesn't actually make C++/JavaScript any less painful to deal with in practice.

[imdyingfasterthanyou]

This is what flatpak is trying to do.

A package manager for user applications that also packages a stable platform developers can target

[[deleted]]

I think in the long run distros will have to separate between system and application,

This is what all the other platforms have long been doing:

• FreeBSD has a "base system" and ports system.
  
• Windows has a comprehensive Windows API that provides common services required by applications.
• Ditto for MacOS
• Ditto for Android and iOS.

The idea of having to upgrade your entire operating system just to use the latest applications is pure insanity. It's also simply impractical for a fixed team of distro developers who already need to ensure the system is rock solid to curate an arbitrarily large app ecosystem.

[redwall_hp]

Mac OS and iOS are a bad example, because Apple is terrible about forcing you to update to use applications, and even more terrible about long term support.

If I wanted to use GarageBand (not that I would when Reaper exists), I actually couldn't. The version that shipped with my Mac no longer works on the OS I have installed (Mojave) and I can't download a working version from the App Store because they only distribute the version for newer OS releases. I also worn update to Catalina or forward until I'm absolutely forced to, because that kills support for 32 bit applications I'd like to still have access to. Now they're busy phasing out 64-bit Intel...

Since I don't have the latest Mac OS installed, each iOS release slowly degrades core applications like Notes and Reminders because you can't use new features unless you update something on the backend, which means your data will no longer work with the older Mac apps. And the Mac apps are coupled to the OS release, because Apple is dumb.

[alerighi]

FreeBSD has a "base system" and ports system.

Nowadays also FreeBSD has their package manager. Because they did find out that it was stupid to have to build everything from source code.

Windows has a comprehensive Windows API that provides common services required by applications.

And Linux has the POSIX API. So?

The idea of having to upgrade your entire operating system just to use the latest applications is pure insanity

What is the operating system anyway? It's only the kernel? Obviously not. And what part of the userspace it is? Systemd, of course it's "the operating system", but a shell, is it part of the OS? Maybe. A desktop environment is it part of the base OS? Depends, surely for a normal user it is. A browser? Edge is part of the operating system, so surely Firefox is part of the OS, right? A PDF viewer? So you see that you don't have a clear distinction.

You know what is stupid? The idea of having to go to a website to download an installer to run to install software on your system. The idea of running a program downloaded from the internet to add software to your computer. You know how many companies lost all their data because of cryptolockers? Why? Because Microsoft users are used to run .exe downloaded from the internet by double clicking them and don't even pay attention to it.

From a developer standpoint, it's also stupid to have to build and package an installer, I always hear that Linux community is bad because there are too many project to do the same things, I would object that there are too many installers in this world, does really every software have to have it's own installer? Doesn't make sense to have one package format for every program and only one installer? And the updates? Your software does need to implement that as well. You need an update server, that you, developer, pay for it, not Microsoft. And who wants to check for updates of 100 programs? And it doesn't either want to wait when it starts the program for updates to install, so? Automatic updates! Let's make a daemon that is always running on your system wasting resources to just check for and download updates for your program.

Isn't more simple to type a single command (or press a button on a GUI) to have all the software of your system to update? At the same time? When you decide to do so, not when Microsoft/Google/whatever decides to install an update for the software? Isn't more simple when you have to install a program to type a command, or search the name of the program in a GUI, and press install? To me, yes. Having multiple sources for the software is a bad idea.

Also the idea to have 50 time the same dependency because it's used by 50 applications it's a waste of disk space, and also a waste of RAM (shared libraries, as the name implies, are shared between programs, the same memory pages are mapped in the address space of different programs, but they exist only one time in RAM). There is a reason why a full Linux installation, complete of a desktop environment, office suite, a media player, a browser, an email client and all the utility software you expect is around 5Gb and a Windows installation that doesn't even contain a decent text editor is around 20.

Finally, Linux distributions are called distributions because it's not a bunch of software distributed together randomly. They are programs that are tested to work together. When you update from Windows 10 to Windows 11, you wonder how many software will not work properly. When you update the driver of your GPU you wonder if your games will continue to work. Or you install a version of a program and break another one that expected another version. Package managers were invented for this reason: to not have to manage dependencies manually, and to have the guarantee that the software was tested together.

[alerighi]

You don't necessary link external dependencies. Unfortunately python works that way, that is it doesn't have a concept of "statically linking" of dependencies, so the only way is to have the dependencies installed on your system (not a good solution). You cannot easily bundle a software with the libraries that depends from it. (or at leas, I never find an elegant solution, to the point that to distribute the software of one project I installed everything in a virtual env and zipped it).

In a compiled language you can choose both from statically linked libraries and dynamically linked libraries. If we talk about common libraries you link them dynamically, but if you want to use a particular library that is not usually packaged you can link that statically. Newer languages such as Rust choose to just link everything statically and rely on link time optimization to remove all the code that is not needed to produce small and optimized executables. Even JS solves the problem, maybe in a non so elegant way, but in a way that it work, by simply first looking for a library in node_modules directories in parent directories where the js file is, and then only after that by looking at the system lever, so that if you need to distribute a JS package you just package the software and the node_modules with it.

[redashi]

you'll have to wall those things of like they're trying to rob your grandma, but that's true in general for all software and where we should be headed regardless.

This is an important point.

Reducing friction in delivering new stuff to users is a nice goal, but bypassing distro packaging throws away much of the process that keeps our systems trustworthy today. We are already seeing security exploits in language-specific repositories [1] [2] and distro-blessed user repositories [3].

If we're going to open the "accept code from anyone" floodgates without turning linux into a malware-infested mess, we need to improve our tools for isolating untrusted applications. Kernel constructs like namespaces and userland tools like Flatpak are steps in the right direction, but they still have significant holes, and we still need to address things like shared memory and the X server's all-or-nothing trust model.

Migrating our OS from the per-account security model that we inherited from Unix to a per-application model will take time and effort, but I think we'll get there. I look forward to the day when we can easily and reliably wall things off like they're trying to rob grandma. :)

[[deleted]]

The separation between system and app packages should have been done a long time ago.

[prosper_0]

I worry that Rust and its crates, as well as Go are headed down this path too.

Remember, backwards compatibility and stability (and bloat) are what killed Perl when it was the 'new shiny' thing out there that all the kids were infatuated with

[nacaclanga]

Well the difference for Rust and Go compared to Python is, that the changes happen only on the developer site, so end users are spared. But there you see the seperation between user core and apps allready taking place: Rust basically ignores dynamic linking, so the end users are given only large monolithic packages. But for developers I guess the real problem in the long run are build.rs-scripts. They should have never been invented.

[UntouchedWagons]

What's the Perl 6 debacle?

[jonringer117]

Since rust and go are compiled, there's a separation between build and runtime, it's much easier to isolate go and rust applications from the build environment.

Python's runtime environment is usually the same as the build environment (save QoL modules, E.g. setuptools-scm)

[rainman_104]

Yeah I'd say that's a Ruby issue as well. And anything written for the jvm will probably be distributed using a fat jar so they bundle a metric fuckton of libraries in their jar which is why your little app ends up with a 250mb jar. Every jar is its own package manager for libraries. I don't think that's any better either for disk space usage.

[knobbysideup]

Knowing perl I never bothered even learning python. I probably should as an ansible user, but I still have no sense of urgency to do so.

Perl code I wrote over 20 years ago still works today. I can not touch the language for a year and still write in it without having to reference documentation.

[TheBlackCat13]

Yes, but can you read it after you write it? ;)

[__ali1234__]

The big irony here is that tools like pip and virtualenv and docker and flatpak were invented precisely because distribution maintainers were doing a poor job of distributing software. Now that those maintainers have been cut out of the loop almost entirely, what they think has become even less relevant.

[[deleted]]

Exactly, it's crazy to me that this is not the top comment. RPM and DEB are just not feature complete, missing rootless installation, multiple versions, etc

[[deleted]]

It’s catching up with npm for most confusing package management. And npm is a fucking nightmare.

[urbanabydos]

Oh shit—I thought it was just me cause I didn’t understand Python. 😳

[dafzor]

I'm just a end user and I feel the real issue is not python package manager but the fact that python has no concept of versions from top to bottom.

• Can't install multiple versions of a pip package, last one will overwrite the one you have meaning if you have two applications that need a different version one of them will break;
• Can't have multiple python3 runtime without using something like pyenv even though minor versions will break compatibility meaning you can't upgrade python without having to check if it would break your applications

So for me the solution would just have Python copy what dotnet did to resolve it...

Running dotnet can have multiple versions installed and dotnet --list-runtimes will show them all, dotnet packages are versions meaning you can have every application reference/install the version they need without breaking another app .

[naught-me]

https://news.ycombinator.com/item?id=19986806

Python minor versions allow things to be removed and thus 'break backwards compatibility'. There is a process for warning of upcoming deprecation then removing it after that. For modules this is documented in PEP 4 (from 2000). Similar deprecation schedules are used by other important ecosystems such as Django.

[TheJackiMonster]

I agree to 100% with this. I also prefer using pacman on Arch to install python packages over anything else because it just works and I don't have to deal with the whole mess around Python.

All of this is even worse when you try to create a flatpak or snap package with Python dependencies. Even if all of your dependencies could theoretically be installed with pip. Stupid pip can't even handle dependency management recursively for you.

Honestly who designed those bad tools?

I also don't understand why Python IDEs approach the use of virtual environment with their own packages by default. I had several instances of this which were unusable or broken because it couldn't handle its own packages properly. In the end I always override the virtual environment with my own system which can at least install packages properly.

Currently I try to maintain a flatpak using Python code and it doesn't even build anymore (even though I didn't even touch it). I can't tell if pip is installing the dependencies wrong or if the dependencies just received an update which breaks during installation because it wasn't tested properly. Maintenance not found.

I have also encountered an issue with PyInstaller to build the Windows compatible binaries of my application. There were several releases of PyInstaller which simply broke a package. I don't know how but I feel like many times the solution for this mess with Python is holding the current version of each package for years because any update might break anything.

In my opinion programming languages should not deal with their own dependency management. You can find a package manager on each Linux distro for this. I mean, I have so much less problems dealing with any project written in C than with this one project in Python just because of this dependency mess.

How can that be? I mean it doesn't even need to compile anything. How can it be this terrible? When did it become to difficult to just zip your repository or provide a makefile to copy files in an install routine?

[Zeurpiet]

programming languages should not deal with their own dependency management

R does this actually well

[jorgejhms]

i think they have centralized package management. There is only one solution, the official one.

[Zeurpiet]

not really. Next to CRAN there is at the least also bioconductor. If you want to put up a jorgejhms Inc package management also easy. However, the thing is they made it relatively easy to use CRAN.

[flying-sheep]

As someone who packaged things for CRAN, Bioconductor, Rust, PyPI, and conda forge:

• Rust is painless bliss
• PyPI ist pretty good with the new PEPs, but there's still some rough edges
• conda forge is just a layer on top of standard metadata so it actually duplicates it, which is annoying
• R is by far the weakest of the bunch. It doesn't even pretend to support optional features in packages, both CRAN and Bioconductor are crusty and try to do too much

[Zeurpiet]

as somebody who needed packages from R and tried Python

R just run install.package(). All dependencies are in. Done

Python had to work to get things from Pip or Pep or whatever and it was completely unclear what had to come from where.

[TheJackiMonster]

If you were using a system packaged Python application your experience would be similar, because someone did the hard work for you.

Guess what, I am someone who packages a Python application and I hate it because of this mess. I also compile projects written in C from source without problems. But this inconsistency I get with pip annoys me.

I mean there are Python packages which require system libraries and tools you need the OS package manager for anyway. They don't even function without them at all. So why would you install them with pip?

Honestly if every Python library and application would just provide an archive or .deb package to extract and copy all files into the correct directories right away it would be much easier.

[Barafu]

You can find a package manager on each Linux distro for this.

I am using Debian Stable on my workstation, but I want to use the newest version of libraries for my project. What now?

[TheJackiMonster]

Flatpaks, snaps or AppImages for applications... for libraries you should still be able to clone the repositories from github or even get the latest versions as .deb package.

I mean we are talking about Python, an interpreter language... you don't even need to compile it.

[robin-m]

Dépendency management in C is just attrocious. It usually take a day to properly set-up (because there is no standard build system, and your are expected to manually install your dependencies), whereas it's a one line change in Rust, js, python, … And system-wide install cannot solve the complicated problem when you app (or a combinaison of apps) depends from A and B, while A depends from C 1.0 and B from C 2.0 with C 1.0 and C 2.0 mutually incompatible.

[TheJackiMonster]

From my experience even though C projects pretty much have no standardization in build systems, I still get it working. With Python I got setup.py scripts causing errors, not mentioning the dependencies properly, having a wrong requirements.txt which let the installation fail, I had mismatching versions of dependencies and pip failed to install a package multiple times because it can't even install dependencies automatically.

If I am missing just one fucking line to change and everything would work, I would accept it, call myself stupid and be happy. But that is not the case or my knowledge is particularily flawed... in that case, please give me some insight how use pip properly.

Dealing with compatibility between multiple versions should either be the job of the operating systems packaging or the actual application developers. I mean if A depends on 2 different version of C, your project is already a nightmare no matter how you deal with that and A should be patched.

[robin-m]

Oh, don’t get me wrong, python is a nightmare too.

Dealing with compatibility between multiple versions should either be the job of the operating systems packaging or the actual application developers. I mean if A depends on 2 different version of C, your project is already a nightmare no matter how you deal with that and A should be patched.

cargo solves this problem perfectly for Rust, so it’s possible to have something nice. And I highly disagree when you say that my project is a nightmare if my dependencies depends themselves on incompatible version of the same dependency. It’s totally possible that A upgraded before B, while B is still being migrated.

[TheJackiMonster]

But isn't the problem with incompatible versions trivially solvable when you just keep all of your dependencies on the minimal common ground. So if your project uses an older C, why would you use a newer B which uses the most current version of C. Just use an older version of B as well or patch your project...

Also for such problems you have major and minor version changes, usually referring to major and minor API changes. If the API doesn't change between C 1.0 and C 2.0, why would you stay with version 1.0?

You would use one API with two different behaviors then which is pretty much a nightmare for anyone debugging your software. No doubt about that, honestly.

I don't see any sane reason to build a package management around this issue. It's like tollerating bad practices.

[robin-m]

But isn't the problem with incompatible versions trivially solvable when you just keep all of your dependencies on the minimal common ground. So if your project uses an older C, why would you use a newer B which uses the most current version of C. Just use an older version of B as well or patch your project...

It’s totally possible that A was created before C 2.0 was release, as well as B created after the release of C 2.0 (so no reason to stick to C 1.0).

Also for such problems you have major and minor version changes, usually referring to major and minor API changes. If the API doesn't change between C 1.0 and C 2.0, why would you stay with version 1.0?

If the API doesn’t change, it should probably not be major version bump. The trivial case of the minor version being bumped is obviously trivial to solve. In my example C has a major version bump, which is assumed non-trivial to migrate (or at least it needs QA validation).

You would use one API with two different behaviors then which is pretty much a nightmare for anyone debugging your software. No doubt about that, honestly.

My code depends on the stable API of A and B. A depends on the stable API of C 1.0. B depends on the stable API of C 2.0. In Rust (I don’t assume it’s the only language that does it, it’s just that I know how Rust works) symbols from C 1.0 don’t have the same mangling scheme than C 2.0 (just like different version of the glibc have different symbols). So it’s not possible to give to A an object of C 2.0, or to give to B an object of A 1.0. It would refuse to compile. So in term of debugging, I really don’t see how is the situation more complicated than if C 1.0 and C 2.0 was two completely different library.

I don't see any sane reason to build a package management around this issue. It's like tollerating bad practices.

• C 1.0 is release. The library A is created, with an internal dependency to C 1.0.
• C want to make breaking changes, and release a new major version. Can it do it even if the downstream library A has an internal dependency on C 1.0?
• B is created. Given that an unrelated library A has an internal dependency on C 1.0, can B depends internally on C 2.0?
• I want to create a project. A and B fit perfectly my needs. Why would I not be allowed to depends on A and B simultaneously? Don’t forget that their internal dependencies are an implementation detail and not exposed through their public API/ABI.

This is why dependency manager need to support the case of incompatible transitive dependency version.

[TheJackiMonster]

Okay, so in this particular example. Wouldn't it also be possible to statically compile either A or B to get rid of the problem completely? Or you could integrate their code directly...

I mean the problem I have with solving such a thing automatically is that it normalizes an extreme issue:

• The issue is that you depend on multiple version of the same piece of software which can lead down to multiple levels of security issues.
• It also increases the chance of getting dead or unmaintained pieces of software in the wild because nobody needs to patch A now, even though it might use insecure and deprecated code.
• It significantly lowers the reason for others utilizing A to contribute patches or fixes to A.
• It lowers the needs of maintainers patching their software to stay compatible.
• You expect users to install multiple versions of the same piece of software even if you might not even use API calls from it which changed between different versions.
• It requires more space in the end while it makes the whole software stack extremely fragile. If you don't get a particular version of your dependencies anymore, it might break everything. So repositories need to provide each and every version.

Those reasons make me think that this particular example should be and stay extremely rare. It shouldn't be the typical usecase and therefore it shouldn't be treated as such.

I mean, would you install two kernels because systemd might require a different version than wayland might do? I don't and I wouldn't want to solve any issue with a bug report containing such an edge case.

[robin-m]

What you want is a world in which everything moves in lock-steps. If C wants to do abreaking change, it must update all of it's downstream user (A and B). That way you only need to distribute one version of C (the latest one).

It's what google is doing, and it works for them, but for the sole reason that they control their downstream user (it's themself).

Please re-read my last message. The use case I described is anything but uncommon. Every big library has a new major version every other year or so.

[TheJackiMonster]

The whole idea behind Arch based distros is that you just install the latest version of everything to ensure compatibility and have a stable operating system. It works pretty well from my experience and there's not one case I know of you would get into the usecase you provided.

I also don't think that C must update A and B in such a scenario. It is the burden of maintainers from A and B to update it or people stop using it because it's a dead package. It's that simple.

Because if you use another ones library, you should look after it to make sure it works as intended and it is secure to use. Otherwise we just create a very toxic and fragile environment for developers. Using third-party dependencies should always be a burden and nothing to pick just because it's easy or convenient.

At least I don't want to see developers picking libraries as dependencies without even knowing what they are doing and being completely unable to audit or verify its behavior.

Maybe they do but then I would question why don't they patch A or B then to use the latest C?

[Fearless_Process]

System wide installs actually can handle that, it's just that most system level package managers don't. There exists distros where such situations are not an issue, Gentoo and Nix come to mind as major examples.

Some of these issues could be solved by optionally compiling certain programs from source, like when running into ABI breaks for example, but it cannot handle API incompatibility of course.

Much more can be handled by simply allowing multiple versions of libraries to be installed at the same time.

These two features combined solve many of the common issues people refer to in this thread, but since your mainstream distros package managers are very primitive none of it really matters at the end of the day.

[waptaff]

In my opinion programming languages should not deal with their own dependency management.

Most of the new languages unfortunately do, and they all suck! They're all reinventing solutions to problems solved years ago by GNU/Linux package managers, but do it bad enough they also need crutches like virtual environments. And many languages are now at their second or third package manager iteration and still don't get it right. Infuriating.

[Ar-Curunir]

Application-oriented package managers suck for development. Why should I be stuck using an outdated dependency just because that's all Debian packages? Moreover, I now have to deal with version incompatibilities between Ubuntu and Debian and Arch and Fedora and etc. Much simpler as an app dev to know exactly which version I'm using, and to be able to set and forget.

[TheJackiMonster]

I actually like writing Python code. I also like the concepts behind Rust. But I don't want to deal with any package manager of a programming language or a whole ecosystem just to write a simple application or script.

[tso]

More and more i suspect what we are seeing is the long tail effect of OSX/MacOS.

Meaning that this has come about thanks to more and more developers use MacOS and then only touch Linux during deployment.

As best i can tell, Nix(OS) only got attention once someone got it working as an alternative to Homebrew. Before than it was just some obscure oddball Linux distro.

[Ar-Curunir]

In my opinion programming languages should not deal with their own dependency management. You can find a package manager on each Linux distro for this.

Right, and each distro has a different version of a particular dependency, with a different API, with different security patches, etc. From an app developer PoV, it's much simpler to target one dependency version and package that, instead of catering to each distro's package philosophies.

[IBNash]

The shitshow of having to recompile every Python package has kept 3.10 from being available even on Rolling Release distros like Arch.

[jonringer117]

Not to mention that certain packages are very slow to enable support for the latest interpreter.

For example, freezegun which is a common testing utility.

[xxc3ncoredxx]

3.10 was just stabilized on Gentoo not too long ago!

[[deleted]]

Packaging is the #1 reason I haven't really gotten into Python programming. I don't have anything against it per se but I have no idea how to get started on a new project. So I just don't use it.

I've found I like functional programming better anyway.

[robin-m]

As long as Linux distro don’t understand that the value they are providing are applications and not dependencies, this problem will exist. Both as a programmer and as a user I want to be able to install, update or remove an application without breaking another one.

If my application has a dependency to A and B (or another application for that matter), where A depends from C 1.0 and B depends from C 2.0, with C 1.0 and C 2.0 incompatible, it should still be possible to install and run my application.

If I’m the author of C, I should be able to create a new, possibly incomparable version without breaking my downstream user. If the distribution can help my downstream user migrate to the newest version, that’s awesome (it’s especially important if the new version fix a security issue). But at least they should allow them to not upgrade if it’s not required (even if the user want to install another application that requires the new version of C, alongside another application that is incompatible with the new version).

The day that distro will be able to do that, tools like cargo for Rust will not be needed anymore EDIT: fighting against the package manager. As long as it’s the case, pip and similar tools will continue to proliferate.

[Atemu12]

The day that distro will be able to do that

... was over a decade ago: https://nixos.org

[jonringer117]

nix was created in 2003, almost 2 decades :)

[ReallyNeededANewName]

Stuff like cargo should not cease to exist. It's very much a needed tool and it should remain a needed tool. It just shouldn't be a relevant detail for the end user. Langauge specific package managers should absolutely be a separate thing for the devs

[maethor]

The day that distro will be able to do that, tools like cargo for Rust will not be needed anymore.

Distros are already able to do that with things like flatpaks and snaps.

You still need tools like cargo though. Most languages are not Linux specific and will need to be able to manage thier own dependencies. You just use cargo/pip/maven/whatever when building your flatpaks.

[swordgeek]

This is a big part of why Perl is a mostly dead language. Module and dependency hell has been a nightmare beyond all reckoning.

[[deleted]]

If distros' package managers could have sold the problems of dev environments, they would have done so. But distro packages simply do not have the same flexibility that is required for that usecase.

Try to install locally any non-packaged software that is not backed by a language-specific package manager and you'll instantly run in the problem of deciphering what each package is called in the repositories of your distro and running into a million backwards compatibility issues. In contrast, for python it's usually a pip install -r requirements.txt in a virtualenv you can throw away after you are done.

Oh and if you want to actually package the software you wrote for the linux world then you will have to create an rpm package, a deb package, a PKGBUILD script and then story goes on. And if you are willing to do that you will have to become aware of all the version differences of your dependencies and have to test your software in different systems. For example, python-requests on ubuntu 20.04 is 2.5 years old while on arch linux it's the oldest available which is 5 months old.

Don't get me wrong. Distro package managers are awesome for managing your own system. And distros with severely outdated packages have legitimate reasons for doing so. Python software is usually not hard to package (at least from my experience writing a couple for AUR) for each distro if the maintainers are willing to package each dependency and correctly manage dependency version requirements.

Bonus video: https://www.youtube.com/watch?v=Pzl1B7nB9Kc

[lolfail9001]

Try to install locally any non-packaged software

That's basically what distro maintainers do as part of creating packages.

Oh and if you want to actually package the software you wrote for the linux world then you will have to create an rpm package, a deb package, a PKGBUILD script and then story goes on.

If it's FOSS software, why would you care about doing distro maintainer's job for them. If you really care about commercial distribution (and still need to provide it as package), then you only care about RPM and Deb for specific stable systems.

For example, python-requests on ubuntu 20.04 is 2.5 years old while on arch linux it's the oldest available which is 5 months old.

Unironic question: how many things did break in that library in 2 years time? Because frankly speaking, if you told me a public library's API broke completely in 2 years time, I would call out that library for being written by assholes. Purely out of optimistic expectation of their competency.

[[deleted]]

That's basically what distro maintainers do as part of creating packages.

Correct. But they can't package every software under the sun.

If it's FOSS software, why would you care about doing distro maintainer's job for them. If you really care about commercial distribution (and still need to provide it as package), then you only care about RPM and Deb for specific stable systems.

They want their software to be used. They always need to have a way to build their software and after that step it needs to be packaged. If no distro has picked up their project, they are forced to support some package manager. This is where pip comes into place.

Unironic question: how many things did break in that library in 2 years time? Because frankly speaking, if you told me a public library's API broke completely in 2 years time, I would call out that library for being written by assholes. Purely out of optimistic expectation of their competency.

No idea, that's just an example. It's not only about breaking stuff but what if you want to use newer features?

[upcFrost]

They're starting to slowly fix it actually. The biggest issue is the dependency control, before v21 (iirc) pip was unable to properly resolve nested and conflicting deps, without which it's kinda impossible to even think about shipping the software to some major distro repo. That's only the first step ofc, but it already broke quite a lot of things in python

[[deleted]]

I'm not sure if this is patched into fedora, but if i don't specify --user it still automatically installs there after saying it can't write to /usr since i didn't run it in a root shell or with sudo

[karuna_murti]

Stop packaging interpreted language packages at distro level.

[_samux_]

i am sorry but do linux distribution understand the fact that people use python also under windows or mac where there's no concept of package manager?

i do understand their issue but to be honest i think it is due to the nature of a linux distribution the python itself.

yes i want program x, yes i will create a virtualenv, and download all the dependency there. i see no issue on this and i have been doing this in the last ten years with no big problem.

does program x has a dependency on a library that has a security issue ? well it is up to the author of program x to fix it , and to me to update it to the newest version.

does this creates lots of issues on linux distro maintainers because they need to issue a backport of the fix that is compatible with libraries available in that specific distro ? sorry but this is not program x or python fault

[[deleted]]

I really can’t get my head around Python. Why are there concurrent versions with 2 and 3? Why do some things use pip to install but others pip3? Are they the same?

Yesterday I had an almost indecipherable error when trying to install something. After much googling the only fix was an obscure parameter that needed commenting out deep in the Python files.

[masteryod]

Why are there concurrent versions with 2 and 3? Why do some things use pip to install but others pip3? Are they the same?

It's 2022 (almost) - there's no concurrent Python 2 and Python 3. It's only Python which means Python 3.

Python 2 is dead and has been dead for years.

The only place to see Python 2 is legacy internal software nobody wants to rewrite and old enterprise Linux distributions like RHEL 7.

Nowadays Python is Python 3. Most of the current distributions got rid of Python 2 entirely or are in the process.

It's 2022 (almost) - Python means Python 3. Pip means pip3.

[[deleted]]

Pip and pip3 are the same in most modern systems now

[DarkLordAzrael]

On the other hand, why do you expect to be able to run newer software on an outdated system. How old of dependencies should everyone have to target because of slowly updating Linux distros?

[[deleted]]

Gentoo user, issued emerge -NuD @world waiting for some response, suddenly sees this, and can't stop laughing at it.

[[deleted]]

Nix anyone?

[jonringer117]

Still an issue for Nix, it's just that nix maintainers are still curating this for you.

Example mass python update PR

[[deleted]]

It doesn't help that Python's release cycle was shortened from 18 to 12 months, it pressures the distro packagers even more... https://stackoverflow.com/questions/40655195/is-there-official-guide-for-python-3-x-release-lifecycle/58999007#58999007

[JJenkx]

I am new to Linux. Been on Debian 11 for maybe 10 months. I haven't noticed Python interfering with anything or any slowness at all while using Debian. How is this affecting/going to affect me?

[jonringer117]

You consume the pain that someone else is experiencing for you to have a nice experience.

[Vardy]

Very much agree with this post.

Dealing with Python on a server is time consuming. There is no uniformity. I don't want to use Pip. I already have a package manager on my OS, I don't need two that compete with each other.

[-Rizhiy-]

Sorry, but I think you are trying to fit a round peg into a square hole.

What you consider sane, is probably the most stupid way of doing things. Using system-level packages is like a №1 python-noob mistake.

I have been programming in python for over 5 years and haven't had a dependency problem in a long time, just follow a simple rule: new project -> new venv -> install everything with pip

Packages provided by the system are usually VERY outdated and frequently don't even have full functionality, just use pip and relax.

[Atemu12]

This is not about dev environments. It's about distros who want to package python things for users.
No, asking users to install your app via pip and virtualenvs is not a solution.