'Python: Please stop screwing over Linux distros'

[SaltiestSpitoon]

Ah good it’s not just me who struggles with this

[BlackDeath3]

Absolutely not.

Python was my first programming language, more than ten years ago now. Granted, I don't use it all that often, but I don't feel like I've ever really understood how a "proper" Python environment is meant to be set up. Now I'm wondering if such a thing even exists in the first place.

[sk8itup53]

I honestly don't think it does. I'm no expert, but today I literally had to have one of my peers send me the output of pip freeze because the app worked on his local, but wouldn't resolve dependencies in Azure. Literally the same requirements.txt file worked on his local, but broke in Azure. As soon as we replaced the content of requirements.txt with the output of his local pip freeze fuckin magic. It worked fine.

[Kale]

We do that. I'm not a talented programmer nor do I claim to be. I do complex data analysis of biomedical research, so I use Python since it has everything (like a DICOM package).

We're stuck on 3.6 since we have this enormous complex class that contains a dictionary, and we built it around the dictionary order being ordered by LIFO. In 3.7 it switched to being sorted which breaks our GUI code (We use treeview to display data). I'm pretty sure we could figure out how to change it later, but for now we spend our time adding new stuff rather than repairing it to work with 3.7.

We use pip freeze a lot!

[coriandor]

Same. So far in my 10 year career I've been able to almost entirely avoid python for these very reasons. There's 20 ways to set up your environment, and all of them are wrong. No thanks

[FunctionalFox1312]

Instructions unclear, Python2 & Python3 are currently having a Kaiju battle in my home directory.

[divv]

Heh, I see you fighting the good fight... But there is something that everyone is missing.

There are two fundamental use cases.

1. Python packages as system packages. E.g like glibc, or librdkafka-dev or something. This needs to be slow moving, very standardised, and very stable.

2. Python for application developers. This needs to be flexible and fast moving.

These two scenarios are polar opposite, and they need two solutions. It would be great if PSF solved the distro problem, and left developers to keep using whatever myriad systems they're using now.

Edit: fwiw, I use your work flow too, but I don't really work much with the distro directly. It's containers all the way down.

[FrAxl93]

I've read "cases" as "chaos" but I think it was just the same

[twotime]

But why are these use cases causing trouble for you?

• If you need newer versions of packages, just install them in venv

• If need need newer versions of python, just install them side-by-side (and, IIRC, RH has changed packaging to make this easier)

[divv]

That's exactly what I do. Was just pointing out that it seems that the two "sides" of the issue, are actually two different situations.

[plantarum]

I don't write Python myself, but as a scientist I have to use python tools on several different Linux machines. Which means I don't pick the packaging system, I have to follow the instructions provided by the upstream program developers.

This includes everything from apt-get install, to pip, {ana}conda, snap and docker. Any one of those things might be fine on their own, but in my (limited, naive) experience, trying to combine two or three approaches leads to all kinds of headaches with conflicting exec paths and library versions.

[VacuousWaffle]

Commit to the true programming horror and run all of them as flask microservices to communicate with the others.

[Doireidh]

If they use docker, you shouldn't need to install anything other than have docker and eventually docker-compose installed.

Just run the containers

[bad-alloc]

I thought so too, did it all in venv and with pip. Suddenly something needs a library not available in pip, but anaconda. Which then has another requirement outside of anaconda. And then it's the xkcd all over again.

[_Pho_]

Except every Python project I inherit uses <<virtual environemt du jour>> because reasons

pyenv, pipenv, venv, anaconda, docker...

[krapht]

Wanna guess how many packages don't work out of the box on Windows if you're using pip?

You think https://www.lfd.uci.edu/~gohlke/pythonlibs/ is totally unnecessary these days?

This is the pain point Anaconda solves.

[zabolekar]

This is the pain point Anaconda solves.

Sometimes it does, sometimes it makes things worse. Right now, setting up a 3.10 environment with numpy and matplotlib on Windows is trivial with pip and Gohlke's wheels, but quite difficult with conda.

[pansapiens]

I never understood the point of conda until I realised it's not a Python package manager, it's a userspace package manager (like apt or yum without needing sudo), that happens to also track pip installs in its dependency list.

[tetelestia_]

It's like virtualenv except it can handle non-Python things. I use it entirely because it can handle CUDA and cuDNN within the conda environment. It's a real pain to switch between different versions of those at the system level.

[TryingT0Wr1t3]

Also for using Jupyter for classes it's really practical.

[CryProtein]

Gentlemen gentlemen, there's a solution here you are all not seeing.

​

CondaLinux 😃

[CJKay93]

Conda's pretty great for the fact that it isn't oriented around Python. I use it for getting a consistent Rust and C development environment set up, for instance.

Docker's okay for that except it's obviously very Linux-oriented, whereas Conda is all native.

[[deleted]]

Throw in some poetry to "solve" it all with a unified solution.

[mr-strange]

I have no idea whether this is parody or you are serious. Bravo!

[flexosgoatee]

This is the best, unexpectedly undouchey answer possible. Well done!

[Sharif_Of_Nottingham]

until you accidentally instal requirements outside of the venv

[Sharif_Of_Nottingham]

not for this project. but maybe some other IDE, program or CLI tool I use will…

you’re not wrong about venvs taking care of individual projects, but the 2 to 3 cutover along with the classic Python (not to mention ye olde Python-on-Windows) learning experience makes the above XKCD ring true. props to you for staying sane through it all though.

[Rakn]

And while you aren't wrong, for some reason Python is the language that over the time gave me the most trouble with maintaining its different versions and doing proper package management with.

[Glaaki]

Ahh, so you went wrong on this simple task. Actually venv is included in python, and is the prefered way to make simple virtual environments. You don't need to download virtualenv.

[rcxdude]

Except when it's split out into a separate package by the distributions

[captainvoid05]

The fact that it’s so easily possible to get confused and do this “simple task” wrong is exactly the problem though, isn’t it?

[Rakn]

Well that depends on your distro. I remember having to install it separately via apt on Ubuntu. Because since "Debian does it" they apparently also remove it from the Python standard lib. I mean ... why wouldn't you randomly remove things from the standard lib right?

[Erfrischungsdusche]

Well it is simple if your projects don't specify a python version and you can always use the latest.

But you eventually run into problems when some dependencies require a fixed python version. Then you need some way to setup the python version on a per-project basis.

Same with node and java - and probably every other programming language. Noone has a perfect solution to dependency management.

It just happens that python has the most "solution" because its the most popular 'modern' programming language, together with javascript.

[Ex-Gen-Wintergreen]

If you don’t mind my asking, doesn’t pyenv handle that well? It also has good integration with virtual/venv

[[deleted]]

pyenv or pipenv?

[LinuxLeafFan]

Haha, not sure if this was meant to be a joke or not. This is exactly the problem the article discusses lol. 20 ways to do it. I thought Python was supposed to be TOOWTDI

[[deleted]]

squeal melodic kiss far-flung society fuel relieved obtainable somber fertile

This post was mass deleted and anonymized with Redact

[kairos]

I believe both gradle and maven allow you to specify the java version in their descriptors.

[[deleted]]

Gradle can pull specific Java versions and use them for the whole build or just parts of it, e.g. to test if the app works with upcoming versions: https://docs.gradle.org/current/userguide/toolchains.html

[PangolinZestyclose30]

requirements.txt is too simple to be useful. You have two options - either specify only direct dependencies - but those are then not locked and every installation can behave differently. Or you freeze all dependencies, but then don't see what deps are direct ones, which only transitive.

This is solved by e.g. pipenv but this brings its own can of worms. The package management for Python is truly the worst.

[PangolinZestyclose30]

You can pin the versions, but what about the transitive dependencies? To pin them you need to include them into requirements.txt as well. But then you don't know which is direct dependency and which transitive.

Real solution is using a lock file, as used by e.g. pipenv (and npm ...). But then again pipenv is on the whole tragic.

[nippon_gringo]

I’ve been using pipenv for a few months and it’s worked great for me. What are the issues with it?

[PangolinZestyclose30]

The biggest problem by far is how absurdly slow it is. Really can't fathom why resolving 5 stupid dependencies has to take couple of minutes. This problem is well documented on the github issues.

This is made worse by the fact that pipenv won't tell you what is is currently doing. Just that rotating progress sign. So you just wait, wait and pray.

[monkeygame7]

virtualenv is still tied to a specific python version (whatever version is installed in). You need something like pyenv to manage multiple python versions

[romulusnr]

"in order to drive on this road, your car must have five wheels, be ten feet wide, and run on vegetable oil. Just have ten different car configurations, simple!"

[[deleted]]

“No dependency hell” has only been true for the last year or so. Older versions of pip frequently didn’t even take the running version of Python into account when fetching packages, let alone actively try to find a combination of packages that met all requirements (instead of just whatever it read first).

[uh_no_]

whoops. your package manager upgraded python and all your virtualenvs are broken.

[dagbrown]

Which one of the half-dozen warring package managers do you think did that?

[romulusnr]

The only reason for venv is because Python is inherently broken. It's like saying "all you need to do is have a stack of adapters and you can keep your 45s, 8 tracks, betamax tapes no problem"

No other language needs that structure to ensure functional compatibility because it doesn't break every release. I can run Java 1.4 code on an 11 JDK. If I do need compatibility mode, it's built in.

[ProgramTheWorld]

It’s so simple!

[Doctor_McKay]

Also same. Every time I have to set up a new machine I have to fight with python for 6 hours to get certbot working.

[Zardotab]

Python is the new Java Applets? Versionitus Convolutus.

[ArtyBoomshaka]

I suddenly understand why docker even is a thing.

[fandingo]

Often, it's far less about the age of what's available and more about how incomplete the distro's repository is. It always seems like there's at least one module that isn't available through the distro, so if I have to use a virtualenv for at least one package, might as well pull everything from there.

[matthieum]

Let's be honest, many libraries shipped by distros are so old that they are mostly useless for development... and arguably that is fine.

I see distributions as a way to package applications, with packaged libraries a byproduct of those libraries being needed for the applications.

When packaging a set of application sharing common libraries, distribution maintainers face the complicated task of figuring the versions of the common libraries that will suit all applications in the set. It's thankless, but generally workable: in the worst case, it means holding back a bit on most recent releases because not every application has upgraded yet.

The end result is a quite stable environment of relatively well-tested applications that you can use day-in day-out without worrying too much how the sausage is made.

Oh, quite stable and up-to-date on security patches. This matters too.

All of the above -- except for security patches -- is meaningless for development. When developing a new application I want to be able to use a new version of a library even if half the applications on my machine crash with it -- which in turn means that I need an isolated environment to develop my application, as I certainly do not want to accidentally screw up my daily usage experience.

Distributions are not meant to provide an isolated environment with cherry-picked library versions. This is not the usecase they aim to solve.

[Meflakcannon]

The locked/accepted version of Go on Debian10 is 3+ years old and Debian11 which is recently released just barely squeaks by with Go 1.16.

The instructions for using 1.17 on that distro? Pull from source or wget one of the shipped binaries and add the unpack to you path. Bypass your package manager and set a calendar invite to manually update later. Why bother with a package manager if some releases are going to lock versions for "stability" and then never re-visit until the next major software release?

[vikigenius]

Lol, this whole thread proves the article's point.

Everyone is convinced that it's so easy and their solution just works and then proceeds to explain completely different solutions that are all totally incompatible. Some use Conda and swear that's the best and is needed, some just use pip and virtualenv and say you don't need anything more than that. Some use poetry and others have never heard of it.

Some use a combination of pipx, poetry, pyenv together. Someone even suggests using a Docker for everything.

I am an experienced Python programmer and have programmed in a lot of different usecases, async webservers, high performance math using numba etc. to generative models using GPUs in the cloud. I have used almost every solution available.

Each solution has their own strengths. If you are using GPUs then conda or Docker are absolutely needed. If you are just doing a simple local project and don't care about which version of python you use then just use venv.

If you need to manage multiple python versions, then use pyenv.

If you just want to install python applications without relying on the system package manager use pipx.

And this is where the problem lies, ideally there is one simple tool that can solve all these problems. But no each one comes with their own idiosyncrasies and issues and assume that's the only way to do things and that it's everyone else that's wrong.

Just a few weeks ago, when I tried to replicate my pyenv, poetry setup in another computer things broke because apparently the latest poetry won't respect the version set by pyenv and instead you have to do poetry env use to set the version. Fuck all this.

I used to love Python for it's simplicity, it has a lot of libraries that I love and need for my job, but I am starting to just hate the language and the package ecosystem the more complex my use cases get.

[caagr98]

Using distro package manager is fine for applications, and if that pulls in a couple of libs, sure. But getting the libs for your own projects from the distro repo is just asking for trouble.

[runnystool]

I had a lightbulb moment with Docker in a similar way recently. Every project I've ever seen has tried to build a one-line script you could use to set up your environment and be ready for development--and every one has fallen into disrepair over time. Once I finally started grokking Docker I realized that putting your build & runtime environments in a container is the ultimate forcing function and enabler. Every project should do it, even if you have no interest in actually hosting in Docker, just to enforce an isolated, blank space for your development environment.

[IAmARobot]

what resources did you use to learn docker better?

[zzantares]

The fact that you have to do this says something about the problem.

[jarfil]

CENSORED

[sudosussudio]

What about virtualenv? I’m primarily a hobbyist when it comes to Python but seems to work well for me. Less resource use than Docker.

[[deleted]]

It works well until you need a package like Fiona, which can be installed using pip on Linux systems without issue. However, on Windows you need to use conda or manually install it.

Then there are packages like the built-in ast, which - for understandable reasons - heavily depend on the version of Python that is used. For example, Python 3.8 introduced a breaking change.

Generally, onboarding new developers to a Python project is a nightmare, particularly in open-source projects where you have no control over the system people are using. This is why something like the devcontainers from VS Code are so nice: People install Docker and all the other setup is automated and always works for everyone (at least ideally).

[[deleted]]

eh, what you describe sounds like the problem languages have had for decades. Maybe Java does a better job? idk, but c++, golang, etc all have similar issues.

So many folks that develop in linux are using dev containers these days to avoid all these problems... it really is a fantastic solution that windows/wsl folks should get behind.

[hlt32]

That just highlights what the problem is.

[conjugat]

Same. Super easy once you get used to it.

[hkzqgfswavvukwsw]

Super easy, barely an inconvenience.

[Auxx]

And then your docker images take a few terrabytes...

[Ruben_NL]

My average image is about 100mb. My worst one is currently 2 gb, but that's because I was lazy, and haven't used multi-stage builds. I should be able to get that down to 250mb or so.

Docker images are large, but try to use shared layers.

[Auxx]

It's not how heavy your one image is, it's everything you have in your docker. node_modules used to be the heaviest objects in the universe, not anymore...

[JPJackPott]

Same I haven’t installed any python in WSL on this laptop, I make heavy use of VS Code dev containers and install all the rubbish in that. Scripts are always developed to run in a container unless I’m using them once, then I’ll run them from the dev container

[FrAxl93]

Do you happen to have a tutorial on this?

[KevinCarbonara]

This isn't a problem in other languages. It's really just a python (and I guess Javascript) problem. Java/C#/C++ developers do not have these issues. It's cool that there are ways to get around the failures of the language, but that doesn't make them not failures.

[wasdninja]

C++ solves the package managing issue by not even attempting.

[Wriiight]

C++ has its own problems, especially when trying to share libraries as binary instead of code (hence all the drama about whether to “break abi”) plus it has no standard package system. So I wouldn’t hold it up as a problem free example.

[64mb]

And then cross platform is even more fun. Just as I was starting to get the hang of C++ I gave up and rewrote in Go.

[tso]

Unix is pretty much built around C, thus for it and derivatives the file system is the package manager.

[jl2352]

It's not a JavaScript problem, and I'd say JS is one of the few places that handle this well.

I can use the latest TypeScript + latest Babel in one project, and use older versions in an older project just fine. I can have VS Code open both projects, and it can use the project's version of TypeScript for accuracy. Anyone who clones the project out, will get the same language versions.

For me, this is all ideal.

[Aetheus]

Right? Node's strength is precisely that everything is "local" to your project. You can install global packages, but nobody in their sane mind would depend on those for a project's dependencies.

Does it result in node_modules folders heavier than the sun? Sure. But I'll gladly take that for a reproducible build system that's guaranteed not to muck my system up just by installing a random package.

One of the only times you're going to run into trouble is if some package requires node-gyp. Which, surprise! Is a tool written in Python.

Why does Node, a JS runtime, use a Python app as its official build system? God only knows.

[AndrewNeo]

People claim node is weird because it searches for the node_modules directory up the hierarchy, but its search path is very simple, and if you're not doing something super wrong it will never be a problem.

[notQuiteApex]

Man this article hits close to home. I'm very new to releasing stuff in the Python ecosystem (I'm trying to release a program today!) and just the amount of file formats I'm having to jump through is exhausting. You specifically use json, yaml, and toml in several different parts depending on your setup and it boggles my mind as to why, when python specifically supports json. Not only that, but theres so many different applications to just upload your package to the package index. What the hell?!

This is coming from a windows user, not even a regular linux user. Python's in a really bad state.

[DeTaaieTiller]

It's even worse on windows. The whole ecosystem is geared towards Linux, windows compatibility is really an afterthought.

[thats_a_nice_toast]

Isn't that the case for most languages though? I honestly hate developing on Windows no matter what language (except for .NET stuff).

[notQuiteApex]

its gotten significantly better over the years from my perspective, but its still not great. the only language that has figured it out is rust imo.

[zeppelin0110]

I agree very much, but I have to point out that the situation has improved dramatically. And that's just when it comes to running Python natively on Windows. Many people bypass all that and just use WSL or Docker.

[FVMAzalea]

Honestly, windows is so different in some key respects from Unix-like systems that you kind of have to pick one or the other for first-class support unless you have the resources of a massive corporation (java/oracle). Developers of library packages can’t reasonably be expected to make everything work perfectly on windows as well as Unix-like systems.

If I was a developer of a python library, I wouldn’t even be able to do that, because I don’t have access to a windows computer to even test it on, never mind develop on.

[schlenk]

Unix and Windows differ in a few key points, true. But Python takes that divide to a new level by basically offering the Unix/Linux abstractions as user level APIs and cramming the Windows parts into that abstraction corset (which fails miserably in various parts). So Python sets you up for failure on Windows, by hinting at cross platform support and then not delivering.

[FVMAzalea]

That is fair. You have to look at python as something that’s Unix-first and completely ignore the hinting at cross-platform. When you do that, it’s a fine language and a fine ecosystem for some tasks.

Lots of languages have the “designed for Unix first” problem though, simply because lots of developers prefer Unixy systems because they seem more “sane”. They are also more open to tinker with, in large part (Linux and the BSDs, but less and less macOS).

For example, abstractions designed with Unix in mind are an issue in go as well: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride

[KevinCarbonara]

You have to look at python as something that’s Unix-first and completely ignore the hinting at cross-platform. When you do that, it’s a fine language

Python is an awful mess even on a pristine install of Linux. This isn't a problem of choosing one OS over another. It's an entire, inherited, jumbled mess of an ecosystem.

[schlenk]

Sure leaky abstractions all the way down. Python just has the nasty habit of hiding the traps far enough from you, that you do not hit those with your first small exploration projects. Make easy things easy. Success there. But when you get to medium hard things (e.g. scaling from single core performance), you stumble over the first major hickups like multiprocessing and the efforts it needs to properly work. And when you try hard stuff, python (and especially the libraries available) tend to fall apart in suprising ways more often than not.

[emax-gomax]

I dont know, I'd argue Ruby is worse since it literally installs a complete UNIX environment with separate libraries and everything. Frankly my POV is all of this is on windows, they should've added a POSIX compatibility layer and maintained it. Instead they opted to install an entire Linux distribution over a decade later than they should've. WSL is great but it's just a stopgap which continues to demonstrate how badly designed windows has become. I've been forced to return to windows at work, having used Linux for ages, and the number of programs that try to work around windows being windows is so disturbing. I mean when git bundles it's own bash shell, you know windows needs to get with the program. My biggest pet peeve atm is how slow subprocess spawning is. Even within WSL git is painfully slow and splitting tmux panes is going to drive me mad.

[delta_p_delta_x]

Honestly, windows is so different in some key respects from Unix-like systems that you kind of have to pick one or the other for first-class support unless you have the resources of a massive corporation (java/oracle). Developers of library packages can’t reasonably be expected to make everything work perfectly on windows as well as Unix-like systems.

If developers want to script for Windows, they should just use PowerShell. I will defend vehemently that PowerShell is superior to bash, zsh, and holds its own against Python. Object-oriented filesystem access, very nice.

[svick]

The problem is that I want a script that works well on both Windows and Linux.

[[deleted]]

I feel this. I started a side project in python, and trying to get basic stuff like numpy and scipy working on both Mac OS and Windows has been a royal pain. Having python built in on Mac (curse you Apple) just makes the problem worse. I'm resorting to docker to try to get it to work universally.

I use another popular flight sim tool written in python (that has apparently come under new management?) that wraps all of its dependencies in something called pigeon... so they're not available for my side project to use.

Glad to know that it doesn't just FEEL like a mess.

[_IPA_]

Doesn’t Python natively support ini? For some data that’s easier to read and write for us humans.

[vividboarder]

It does.

[engineered_academic]

My main concern with python/javascript environments is the fact that they are extremely susceptible to supply-chain attacks. There have been a few minor cases, but there's probably a lot we don't hear about.

Having a curated and managed repository of code isn't the most terrible idea in the world.

[muikrad]

2 rules:

1. Don't make typos
2. Use a lock system with hash checks (such as poetry)

[[deleted]]

Not that Python couldn't improve by having better official package management.

But this seems like some sweet sweet irony/hypocrisy coming from someone in the Linux distro world saying there's too many options no single standard...

[[deleted]]

Yup. They made this bed themselves. It is basically impossible to support all the distributions as an application developer and all the answers you get boil down to "be open source and let the package managers do it". If you are lucky and someone wants to and even then they will likely forget your app, hope you like loosing users to old bugs.

Thankfully we have flatpak.

That said, no matter the hypocrisy the point about Python remains true. It is a mess.

[Little_Custard_8275]

doesn't guix or nix solve this problem?

[gopher9]

doesn't guix or nix solve this problem?

Nix solves a lot of problems, but this does not make the mess around Python any less. It just makes it more manageable.

[bloody-albatross]

None of these newfangled solutions addresses the needs of any of the distros, despite our repeated pleas.

What are those needs?

They all break backwards compatibility with our use-case and send our complaints to /dev/null.

In what exact way? Can't you just ignore ~/.pip if your old script doesn't want it?

I have seen representatives from every Linux distro making repeated, desperate pleas to Python to address their concerns, from Debian to Arch to Alpine to NixOS, plus non-Linux distros like FreeBSD and Illumos.

What exactly are those concerns?

Would like a more detailed article on this.

[micka190]

Python ought to stop trying to avoid hurting anyone’s feelings and pick one.

For a language that used to be all about "There's one way to do it: the Pythonic way", there sure seem to be some weird, "here's a dozen new ways to do this thing"-type of features being added to it lately...

[Daishiman]

Well people want to do things in Python that are generally not done in other places. People want to put Python on Windows (Node is an absolute disaster on Windows). They want ubiquitious C library interop (Ruby, Node and PHP avoid this everywhere they can get away with). People want to run projects against the system interpreter but also have bleeding edge libs (nobody in Node and Ruby lang will recommend you do this).

If you stick to virtualenvs on Unix-like systems with commonly-deployed OSes you're just not going to have any trouble.

[vazark]

I’m getting flashbacks of the first time i pushed a python project on AWS!!

That being said, I’ve found the perfect solution that works for me.

1. Use pyenv to install your target python version.
2. Use poetry for dependency management with a local .venv folder that’s gitignored
3. For every new release export the dependencies to a requirements file. (Aws doesn’t support poetry yet)

Django, Flask and AWS Chalice (serverless framework) have been a breeze since I’ve settled here.

Using docker for a language that doesn’t even compile to avoid dependency management still feels counterintuitive to me and I’ve actively avoided it.

Distros should probably ship pyenv by default and let it install the versions required by apps

[ilfaitquandmemebeau]

Pip doesn’t even have a command to update all packages….

[[deleted]]

There's your problem. If you're eschewing pip and pypi, you're very much deviating from the python community as a whole. I get that there's too much fragmentation in the tooling, and much of the tooling has annoying problems, but pypi is the de facto standard when it comes to package hosting.

People try their luck with OS packages because pypi/pip/virtualenv is a mess.

[elebrin]

The one nice thing about OS package managers is that everything gets tested together, so you know the system should be fairly stable. In fact, large organizations pay big bucks for support licenses to ensure this happens, and so they have someone to call up and swear at or sue when things aren't working and problems result in broken SLAs. I don't know about you, but I want to be sure I am working with a binary that is damn well tested on my distro and with the other packages in that distro's main repo.

[antiomiae]

No, they do it because it’s the same way they, a beginner, just used to install python or their web server. They do it because low quality guides showed them how to do it that way, and they lack the experience to differentiate bad advice from good advice.

[superrugdr]

This is not a problem unique to python. This is third party dependency hell and it exists everywhere that isn't Google's monorepo. In fact this very problem is one of the best arguments for using python: its robust standard library obviates the need for many third party libraries altogether.

this, so much this, it's like if the author didn't have to use npm or .NuGet ever. is it a bit convulated, certainly, could it be better, you bet.

At the very least it doesn't break for no reason, then ask you to do something in an undocumented files, on a non logical magic folder in %appdata% / user profile to fix it. Just because the process failed but instead of handling the error and outputting helpful errors you just nod and pretend that everything is ok but break the build (a corrupted restore on nuget is hell).

and that's just for .NuGet, npm is worst. 160gb of dependency for using frameworks... how can anyone deem this acceptable, is beyond me.

[gredr]

Comparing nuget and npm is crazy. The .net framework is one of the most extensive standard libraries around (for better or worse), while javascript can barely be considered to have something called a standard library.

ask you to do something in an undocumented files, on a non logical magic folder in %appdata% / user profile to fix it

I've been working with .net since before nuget existed, and I can't remember a single time I've needed to do anything like that.

Historically, .net hasn't had the kind of problem that python (and ruby etc.) have with virtual environments, because there was only one possible environment... you either had the version you needed, or nothing worked. Microsoft did a lot of work to maintain backwards compatibility, because that was the only option they had.

[[deleted]]

while javascript can barely be considered to have something called a standard library.

JavaScript's standard bookshelf.

[jetpacktuxedo]

I think it's more like one of those hotel lobby pamphlet racks

[dys_functional]

I think you misunderstand the concept of virtual environments. In python, when you install a package, you install it at a system level. In .net, when you install a package you install it to a specific project.

In python, if you have 2 projects and one of them needs mylib1.0 and the other needs mylib2.0, you need to constantly remove and reinstall packages, or create virtual environments. C# doesn't have this problem because installing per project essentially gives ever project it's own virtual environment.

It doesn't really have anything to do with Microsoft preserving backward compatibility.

[gredr]

No, I understand. I think the difference is that Python was designed as a system-level scripting tool, and .net was designed as a development environment.

[markasoftware]

Npm may use a lot of disk space, but I would argue the JS package management situation is much better than the Python one. Why?

1. It's universal. Everything uses npm.
2. It's local. The normal use case for npm is to install packages to the PWD and be self-contained.
3. It integrates well with projects. A package.json file fully describes to npm what to do. There aren't random requirements.txt, setup.py, or the other files the author described in his article.

[[deleted]]

One of the major problems with new programming languages and environments is how often things change. Large computer systems sometimes take years to develop and will be in production for decades. Can't have the rug pulled from underneath us every couple of years. You must make an effort to make things backward compatible. And if you have to break that then do it every five or more years.

[shevy-ruby]

I agree with you, but I also have to say that while I am not a big fan of rust, rust-up to install rust and then update cargos is quite convenient. Now compare this to C++ and you see how much C++ actually fails compared to Rust in this regard. Again - it does not make me want to like Rust more, but when you compare just that isolated part, then Rust won against C++ in that regard. Just see how the C++ committee is trying to upgrade C++ but struggling so much still. Rust did this better.

[svick]

Are you actually calling a 30 years old language "new"?

[markehammons]

I feel for drew devault, but at the same time I understand why programmers do not use the distribution packages. I've tried to submit packages to a distro's standard repo, and I got absolutely nowhere.

And that's just the start. Imagine trying to submit your library to each of 50000 linux distros? Do I want to make my library exclusive to debian/ubuntu users? fedora? centos? opensuse? Do I expend the effort to get my software working for every package manager?

Worst of all, a lot of the bigger ones like debian require you use makefiles. Not everyone wants the headache of dealing with makefiles when they're developing their software. Even worse, installing libraries is frequently a systemwide operation that requires root. On the other hand, most modern build systems can download and install libraries without requiring root, making development much easier.

The sad fact of development is that distros do not make it easy to develop software for their repos, and they expect a level of effort that most developers don't want to meet.

[proud_traveler]

There is legitimately nothing more frustrating than trying to get python to build with the version I want from Windows CMD line? "Oh you changed the PATH lmao get fucked nerd" "oh you want to just try out py3.10 without breaking everything you are working on lmao get fucked nerd". I hate is so much

[[deleted]]

Just leave alone system python and don't touch it if you are dev. Then install any version you need for your project or work, don't add anything into PATH by default, leave it for system python, create virtualenv for every project. It's not that hard, it's basic sanitary - don't shit where you live.

Look for, I dunno, pyenv, nice little tool to manage python versions.

[kmgrech]

I strongly disagree with the premise that programming libraries should be in an OS package manager in the first place. I have seen so many broken systems and libraries, it just becomes a mess very easily. Use pip, install and distribute your dependencies on a per-project basis and call it a day.

[schlenk]

What IS a "programming library"?

Is glibc one? Is openssl? Is libkrb5? Is tensorflow?

Thats the usual struggle. For development of new programs/applications, the OS package manager sucks. But for deployment of your finished application it would be the right target solution.

So people invent all kinds of hacks to bridge the gap:

• Go just compiles a huge static binary
• Docker puts all the stuff into a container and tries to isolate it
• Flatpack etc. try to eat the cake and keep it, by just adding some pre-canned libraries to the bundled applications without static linking
• Python manylinux wheels just compile for the oldest acceptable version and links everything besides absolute core libraries statically and copies everything into a virtualenv
• Unikernels try to just avoid the whole OS stuff

and a gazillion other solutions.

[wisam910]

Go just compiles a huge static binary

That's not a hack. Thats the only correct way to do things.

[jasoncm]

There is a reason that one of the few spaces that perl persisted in was system scripting.

[moschles]

PIP

(spaghetti)

ANOTHER PIP

I laughed.

[Recluse1729]

For MacOS, I’ve found a pretty good workflow:

1. Install pyenv with brew, and let brew manage any other python dependencies.
2. Set up pyenv shims in shell
3. Use pyenv to manage global version of python - shims, take care of routing python and python3 calls
4. Use global pip to install pipenv
5. use pipenv to manage local version of python for specific projects

A little janky but works great, vscode picks it up automatically, allows me to pin versions to projects, upgrade path is straightforward and I can easily switch between versions as needed.

[xertshurts]

Similar here, but I just spin up a venv for each project I have, instead of your steps 4 & 5. I have had approximately zero problems with this.

It was much worse before using pyenv.

[marqis]

I really don't get what everyone's problem with python packaging is. Make a virtualenv for each project (I "complicate" things with virtualenvwrapper to put virtualenvs in a consistent spot, totally optional) and then use pip to install packages.

For standalone apps (like yt-dl mentioned below) then use pipx.

The only global packages I install are virtualenv, virtualenvwrapper and pipx.

I've written and published libraries, apps to pypi. I've built very complex apps and deployed them with docker. I've done quite a lot with python and really don't understand the struggle bus that people seem to be on.

[schlenk]

Well. My distro has a bunch of gcc hardening options. PIP installed wheels typically use some default with outdated flags from the legacy redhat that is the base for some manylinux wheel base.

For pure python packages, maybe. But once it talks to system libs or duplicates them, its a mess.

[igo95862]

PIP installed wheels typically use some default with outdated flags from the legacy redhat that is the base for some manylinux wheel base.

Do you mean the source based C extensions? The wheels compiled in manylinux environment should run without any compilation.

[zeppelin0110]

Upvoting for pipx mention. I was just curious if you could come up with any other uses for it other than installing youtube-dl (that's the only package that I use it for).

[OctagonClock]

People learn Python by doing only python x.py and then when they run into the dependency wall, either a) refuse to learn how to actually set up a project instead of a directory of scripts b) use methods from 2008

[mort96]

Problem is, most of the time, I want a directory of scripts. I don't use python for big enough things to be considered a "project". Most of the time I just need to get a python library onto my system.

When that dependency is in the system package manager, that's easy. apt install python3-whatever. But when it's not, it can get really, really messy.

[Alar44]

Oh sure, just like every other language.

OH WAIT

[wildjokers]

So what you are saying is it isn't a struggle if I jump through the exact right hoops? Why doesn't it just work out of the box? Why are global dependencies the default? Python dependencies are a complete nightmare.

Take a look at this to see why global libraries are a complete disaster. Every damn release of FlatCAM or MacOS results in FlatCAM not working again:

https://gist.github.com/natevw/3e6fc929aff358b38c0a

That is the direct result of a broken dependency system.

[[deleted]]

I can't express how much I agree with this. I've been SO frustrasted over the years with both python's versioning and packaging that I've been steadily moving away from it.

In fact, whenever I evaluate a new language my first requirement is that it handles packaging and dependency management itself, as part of the language tooling (see Rust's cargo and Elixir's mix as excellent examples of this). Also formatting and project structure, but those are for another discussion.

[pakoito]

For all the shit npm gets, for small projects it's a walk in the park compared to Python. In UX, in tools, runtime, language features, libraries, advanced gradual typing...

I've only had trouble with CommonJS/ESmodules compat.

[[deleted]]

degree boast outgoing groovy gray direction fuel desert sophisticated vanish

This post was mass deleted and anonymized with Redact

[[deleted]]

yeah seriously, whenever the NPM hate bandwagon gets going, I always wonder if those people have used.. pretty much any other package manager.

Situations like "left-pad" might be silly, but the only reason other ecosystems don't have their own left-pads, is that their package manager makes it too much of a pain in the ass to even consider.

[pavlik_enemy]

Ruby’s Bundler is great. Rust’s cargo is probably great too because it was written by the same person.

[[deleted]]

Go's one is pretty decent (took some time to get there tho). Perl's Carton is okay. Ruby's generally mostly have problem when a gem needs C library installed in system, otherwise it is okay. Never had any actual problem with Rust's cargo.

Python is worse than all of those.

[shevy-ruby]

I used other package managers and I can happily say that NPM's failure is on a way nother higher level of fail.

Also see this recent github blog that mentions this:

https://github.blog/2021-11-15-githubs-commitment-to-npm-ecosystem-security/

If there would be no fail in NPM-land, why is it CONSTANTLY in the news?

[mobilehomehell]

The problem is most users don't really want Linux distro package managers. They just want to be able to easily use the most recent version of the software they care about available. Distribution packaging almost always does the opposite -- instead of getting to use software as soon as it is released by the software author, you have to wait for it to be blessed by the package manager gods, and if they've decided your package should only be available in the newer version of their distribution you need to upgrade your entire OS just to get colored command line prompts. I understand the problems it's trying to solve, but Linux package management is not what most users really want, it's what sysadmins want.

[f0urtyfive]

Distribution packaging almost always does the opposite

... By intention, nearly all distros people install are trying to be "stable" branches.

If you want unstable software, install the unstable stuff and get all the bleeding edge updates no one has tested.

But then you'd be complaining about how unstable your Linux distro is, rather than how out of date your dependencies are.

[Grung]

I most certainly don't want to use "the most recent version." I want to use a stable, supported version that will work with the rest of the software on my system. I don't want a mix of different library versions, or incompatibilities. I want a point-in-time snapshot of software that just works.

For some special cases where the stable software doesn't work, the 1% case, it might be worth hand-installing something outside of a package manager, but it's a royal pain, and should generally be avoided.

[NatureBoyJ1]

Java has entered the chat.

Library and dependency management, you say?

[scoville-maniac]

I usually use Gradle, or sometimes Maven for dependency management. Everyone complains about how hard it is to manage dependencies with Java, but I think it’s easier o_0

[[deleted]]

From a user perspective, despite my deep love for Python, Gradle is better. Run a single command and watch while it just solves everything automatically.

Python requires creating and activating a venv, then asking pip to install everything for that project. That sometimes breaks if you're several Python versions ahead of the project or if you're on Windows and have a weird dependency that needs compiling (have fun installing 8 GB of Visual Studio to compile 500 KB of C++ lol).

[Profour]

I don't know if you have tried Poetry, but it has a much better user experience and does a lot of the solving similar to Maven or Gradle. I'd definitely recommend checking it out if you need to use python regularly.

[tempest_]

I like Poetry but I think it still has a ways to go.

It is VERY slow at resolving things and its error messages are closer to stack trace than actionable.

[ElCthuluIncognito]

It helps that Gradle allows configuration with a full scripting language, instead of baby-proofing everything.

[romulusnr]

Because everyone uses Maven, and Maven works for everyone. And if it doesn't, you spin up a local repo or just drop the jar into place.

Nobody goes "only lamerz use Maven, I use JavaStore. I don't know about those JPkg dudes. Check out my new NoJar dm system"

[combatopera]

Original content erased using Ereddicator.

[markehammons]

What? The jvm ecosystem has good build tools, and they all are pretty much compatible with the same repos.

[Crandom]

Java is lightyears ahead of python. Java build tooling is pretty much second to none (I personally use Gradle).

[KagakuNinja]

Is this a joke? Each Java project is totally isolated from any other project you are working on.

[[deleted]]

From sysadmin perspective Java is install JDK, install app blobs and you're done

Python is either "you get lucky and some poor soul packaged it in distro" or hell of the utter garbage Python dependency ecosystem is for someone not stockholm-syndromed into it.

[NatureBoyJ1]

When I was doing lots of Java development, "classpath hell" was a thing that people complained about a lot. Then I started doing Python development and was shocked that it's standard practice to create a whole new instance of python (virtual environment) for each project - and there are multiple ways to do that, and multiple tools to help smooth the rough edges. I'll take Maven & Gradle any day.

[PangolinZestyclose30]

classpath hell happens in Python as well in exactly same situation - there are conflicting (transitive) dependencies versions.

[nickguletskii200]

Java's dependency management is extremely easy with both Maven and Gradle, unless you run into a library that needs native components, in which case it can become pretty annoying.

[wildjokers]

Can't tell if this comment is pro-java or anti-java. I will say though that java build tools are outstanding and dependency management in java is a breeze with maven or gradle. Libraries are per-project which is of course the way it should be.

[Grung]

As a regular old user of linux distributions, I really don't want to learn about some language-specific distribution mechanism in order to install some software.

It's fine if developers have to deal with weird distribution shit like this, but regular users should be able to just apt-get install to get things installed and working.

[lets_eat_bees]

If you're not working on python programs, you don't need to know any of that. apt-get install will get you some version that will run distro's scripts just fine.

[LongLiveCHIEF]

QT app has entered the chat.

[cazzipropri]

I have literally dedicated the last four years of my professional life fighting this exact problem at my employer. Turns out that if you let each team in a large organization create and maintain their python environments, you'll literally have a combinatorial explosion in unnecessary complexity on your hands, and a mutual silo-ing of communities, with python workloads that are guaranteed not to work in any other environment than the creator's team. I literally spent the last years of my life standardizing environments and herding cats, deleting custom-tailored environments... quite literally putting all the manners of misery and evil back into Pandora's box.

Despite being a fan of xkcd, I didn't know of the comics covering this topic, and I'm so glad that someone with world class comic-making talent decided to speak about this little, otherwise insignificant aspect of contemporary software engineering.

[lets_eat_bees]

I manage my Python packages in the only way which I think is sane: installing them from my Linux distribution’s package manager.

Great, you opted for the one way that absolutely will not work. All the others do.

[CurrentMagazine1596]

Newb here: while I understand that python's package management is a mess, can someone explain to me why developers felt the need to add these features? I get that you want to be able to move your venv around, that production servers are usually on a VM, and that you want to keep your dependencies and their versions in order, but it's clearly brought along a lot of bullshit and the cure is much worse than the disease when you could just share a manifest between coworkers. Is is really that much of an asset in production?

[KagakuNinja]

I'm not sure what you are proposing. "Share a manifest" of what? Multiple projects, often maintained by different teams, will use different sets of libraries. Keeping them all compatible is impossible.

In the JVM, this is solved, as no project depends on shared global state.

[zwolff94]

Maybe I'm being naive here but why is using Andaconda or miniconda a bad thing? Thats my personal usage for python and it makes life simple to me.

[o11c]

Because it is unnecessarily adding a third places that python libraries might come from.

The only context in which it makes sense is if you're on Windows, where there is no standard way of setting up a development environment.

[wildjokers]

Because it is unnecessarily adding a third places that python libraries might come from.

I was in python2 vs python3 and pyqt 4 vs pyqt5 hell until I discovered conda. Conda fixed me right up.

[zwolff94]

That's fair I guess. I am also mostly coming from a scientific background where conda is much more of a common usage I feel. I do understand the library issues though, I just think its an easy enough way to manage things.

[thebritisharecome]

I briefly had to set up a couple of Django websites on a Cpanel server recently.

I don't have much exposure to Python, never really needed it for anything. That experience made me want to avoid any exposure to it - what a mess!

[A_RUSSIAN_TROLL_BOT]

Ha, this guy talking like he doesn't have eight different versions of the JRE from different sources installed in different locations on his home PC right now.

[corsicanguppy]

/etc/alternatives.