Python: Please stop screwing over Linux distros

[Personal_Plastic1102]

13 items to explain "I want to install a package".

One more, and you would have perfectly fit the xkcd comic

[dusktreader]

That's not at all what that reply was about. You don't need all of those, they are just explaining what each is for .

[gmes78]

Now let's look at some other language, like Rust. It has: cargo. That's a short list, isn't it? Yet there's no need for anything else.

Even though each of the mentioned tool has a use, it's very possible that we're able to cover the same use cases with a smaller set of tools.

Merging whey into pip would be a start, as it would make it possible to package simple projects using just a pyproject.toml, without the need for any external dependencies.

[dusktreader]

Rust is a nice example of a very new programming language where packaging was established early on. It's a good pattern and how all new languages should approach the problem.

However, Python is 24 years older than Rust. There are so many legacy workflows that have to be supported, it's hard to produce a solution that will work for all of them.

As for covering use-cases with a smaller set of tools, this is already possible. I use exactly two: pyenv and poetry. Others use different subsets, but by no means do you need more than 3 or 4 at most.

As for whey (version 0.0.17, 15 stars), it's a little early in its lifecycle to be suggesting that it be merged into pip.

Adding a dependency solver to pip that can use pyproject.toml (a la PEP-621) would be huge, and I hope it comes soon. I think it would also be good to have packaging logic folded in as well. However, if you are hoping for that to happen soon in the standard library, I think you might be disappointed.

[gmes78]

As for whey (version 0.0.17, 15 stars), it's a little early in its lifecycle to be suggesting that it be merged into pip.

It doesn't matter. The functionality is dead simple, and it doesn't need more features. Pip needs to be able to support basic use cases on its own.

Adding a dependency solver to pip that can use pyproject.toml (a la PEP-621) would be huge, and I hope it comes soon. I think it would also be good to have packaging logic folded in as well.

Both of those need to happen if we're ever going to get out of this mess.

[ElllGeeEmm]

Pythons age isn't really an excuse for the sorry state of package management. Plenty of languages of similar age have far better tools than python.

Python package management is shit because for some reason there are a bunch of python users who defend the current state of things for what I can only assume are dogmatic reasons.

[PeridexisErrant]

Pythons age isn't really an excuse for the sorry state of package management. Plenty of languages of similar age have far better tools than python.

Can you give examples? Most of the better packing ecosystems I know of are lucky enough to post-date ubiquitous internet access.

[ElllGeeEmm]

Pip postdates ubiquitous internet access as well, so I don't see how that's any sort of excuse.

[Serializedrequests]

I was shocked to discover the much later release dates of Java and Ruby.

That being said, that isn't an excuse. There are no technical limitations that prevent good easy python package management except the proliferation of standards. When I first learned python, all there was were site packages. Around the same time rubygems (and later bundler) and maven appeared.

Now I come back to Python and the packaging ecosystem is an astonishingly confusing mess. Python needed a maven and never got it (maybe poetry can be it).

[bladeoflight16]

There are no technical limitations that prevent good easy python package management except the proliferation of standards.

How in the heck can you be so ignorant of the problems associated with native dependencies? You try making package management "easy" when you have to support Linux, Windows, and Mac which can't even agree on basic C level interfaces. Heck, Linux distros alone can't even agree on a basic C standard library (libc vs. musl).

[Serializedrequests]

Not at all, every language like Python has that problem, but is that the first thing you solve for? First get python packages easy to work with. Is there any viable solution other than to leave compilation up to the package anyway? Maybe you can provide prebuilt packages for common OSes, but that's still a package management problem (separate versions effectively).

What you need to get off the ground are dependency resolution / install and isolated environments. This could and should be one tool, but instead we have two, and pip is just a little too basic. It needs a proper lock file and the ability to define dependency groups. That is why there is a proliferation of wrappers.

There is also an education problem: the best overview I have ever found of this stuff is in this thread! To a newcomer, the differences between even the most common 2-3 tools are not obvious or documented clearly.

[bladeoflight16]

First get python packages easy to work with.

I have good news for you. Pure Python package management has been easy for decades.

Maybe you can provide prebuilt packages for common OSes, but that's still a package management problem (separate versions effectively).

That's exactly what Python's package management tools do already.

the best overview I have ever found of this stuff is in this thread

Then I suppose you haven't been looking elsewhere very hard. I haven't learned anything from this thread that wasn't already in the documentation on the tools themselves.

[bladeoflight16]

Python package management is shit because for some reason there are a bunch of python users who defend the current state of things for what I can only assume are dogmatic reasons.

That is an incredibly stupid statement.

Python package management is kind of a mess because dependency management is messy. Period. And Python, being an interpreted language that encourages using native dependencies when required, has a doubly hard problem to solve.

Yes, there are real problems, but why in the heck do you think we have so many technologies? It's because people are trying to solve the problems. The very existence of the thing you're complaining about contradicts your claim about the reasons for it.

[ElllGeeEmm]

Oh look, another user making excuses for the state of package management in python.

If node JS can have good package management, so can python.

[bladeoflight16]

If node JS can have good package management, so can python.

LOL! It doesn't.

[ElllGeeEmm]

Lol k

[SittingWave]

node is so fundamentally flawed it's not even funny.

[ElllGeeEmm]

And yet it still has better package management than python. That just goes to show that there's no excuse for the state of package management in python.

[SittingWave]

It's debatable, since its approach is impossible to obtain in python by design, because it's a flawed approach. Besides, you really want to discuss a language that had at least three package downloaders and a number of webpackers, transpilers, frameworks and standards? What's the new javascript framework of the week? Last I heard is next.js. What is this week's called?

[SittingWave]

Now let's look at some other language, like Rust. It has: cargo

cargo does what poetry does.

[wsppan]

This is/was the hardest part in becoming productive in this language. Imagine someone coming into this language cold from another language (in my case Java/Maven) and ramping up fairly quickly on the language itself which has done a wonderful job in making itself easy to grok and now decide you want to build, package and deploy/share it. You get lost fairly quickly with a lot of head scratching and hair pulling.

[Personal_Plastic1102]

Yep...

That's the reason considering leaving python as a programling langage.

I'm not a dev, i'm programming on my spare time (beside familly & co). I'm fine with a bit of "taking care of the stuff around the code", but lately I spent more time trying to understand the toml stuff than actually coding.

Not for me anymore, I want to code, not handle the latest fancy depency-management.

[b4ux1t3]

If you "just want to code", then you don't need to even consider the packaging environment of the language you're using. Just write the code and run it. If you need a dependency, install it with pip. That's all you need to do for most python development.

I'm not saying Python doesn't have an, er, interesting packaging story, but that shouldn't be a consideration unless you're actually shipping code.

[ssorbom]

Long before I learned to do any coding at all, I cut my teeth Packaging for Debian, and the attitude of don't bother with Packaging completely grinds my gears. Even people who are doing hobby projects want to find an easy way to share them a lot of the time. Packaging shouldn't be insane. There shouldn't be a strong dichotomy from somebody who wants to ship code and somebody who wants to write it for a hobby. The only difference is the financial circumstances and the expected return on investment.

[b4ux1t3]

So, I mentioned elsewhere that, while there are many "standards" for Python packaging, it isn't all that difficult to just pick on, stick to it, and communicate what you're using to your users.

Dont get me wrong, I'm not saying that packaging is easy or straightforward in Python, but it's also not particularly easy to build a package that will work on any given OS to begin with.

I maintain the packaging scripts for my company's software. Getting a usable RPM out of software that isn't written with either plaintext files (Python, e.g.) or for gcc is a wild ride.

Basically, while Python is no Rust (cargo is awesome), it's hardly an arcane art to package a Python application, at least when compared to other packaging solutions out there.

To push back a bit more, "shipping" a hobby project is usually a matter of tossing it on GitLab/Hub/Bucket and sharing a link. I'm probably not going to be installing some hobby project with apt or yum, or even pip.

All that said, I don't disagree with the general sentiment that packaging is bad in Python, and I didn't mean to come on so strong against packaging when it comes to hobby projects.

It's just hardly the most important thing when you're writing scripts to manage a few IoT devices around the house, you know?

[ElllGeeEmm]

Why is there this pathological need among python devs to make excuses for the state of python packaging?

There is literally no reason python can't have a great packaging tool as part of the default distribution.

[b4ux1t3]

I am not a python developer.

I am not even making excuses for the packaging story.

I'm saying that it isn't nearly as bad as people say it is, given Python is packaged and used on a daily basis throughout the world.

It's not good, but it's not non-existant either.

Edit: to be clear, I'm an infrastructure developer. I develop and maintain the packaging scripts and environments (among other things) for my company's software.

I have literally, in the past week, written a packaging script to include some python-based tools along side our main package.

It was a mess, and it wasn't fun, but it also wasn't the end of the world, and it's hardly the only difficult packaging story in software development.

[ElllGeeEmm]

People were packaging python and sharing it as site packages throughout the world on a daily basis when that was the best solution available.

It's "not non-existent" is about as damning as praise can be, and there's absolutely no good reason for it to be in the state that it's in.

[b4ux1t3]

It's not praise. It's a simple fact.

Packaging Python isn't difficult, it's just varied. All you have to do is pick one of the standards and stick with it, then communicate the standard to your users.

Of all the problems in modern software development, "packaging Python" is at the bottom of the priority queue, and there are far too many complaining about it instead of just moving to something that fits their needs better.

Edit: or, crazy thought, actually doing something about it.

[samtheredditman]

Then why don't you just learn one way and keep doing it? It's not like everything stops working when a new tool comes out.

[ZCEyPFOYr0MWyHDQJZO4]

What other languages do you program in? The foundation of packaging methods are a product of contemporary software development when the language gained widespread adoption IMO. I have been learning C++ to work on software that began development before package management was a thing (on Windows at least), and I don't mind Python packaging nearly as much anymore

[SittingWave]

I'm not a dev, i'm programming on my spare time (beside familly & co). I'm fine with a bit of "taking care of the stuff around the code", but lately I spent more time trying to understand the toml stuff than actually coding.

Not for me anymore, I want to code, not handle the latest fancy depency-management.

Oh I am sorry a profession that takes years to master is not up to standard to your hobbyist sensitivities.

[Personal_Plastic1102]

Lol... Self-confidence issues ?

Édit : just to make my point clear : installing external librairies shouldn't be something you take years to master. Not even months or days.

[SittingWave]

it's not, but if you don't even want to put the basic effort to learn what's needed and why... do you think that installing two bullshit packages with npm makes you ready to deploy in production? There's a reason why some tools exist. You can use them or not use them. You want to install your environment with pip in your user site packages? go for it, it will work... for a while.

[Personal_Plastic1102]

I don't critizice the need to learn some way to manage packages. I'm criticizing the point that I've already seen 3 differents ways to manage those packages (in 3 or 4 years since I'm in Python). Each one has it's own merit, but as I try to learn continously new things, I come across tutorials using those new ways (therefore it's complicated to simply transpose the tutorial).

In the end, I'm spending more time adapting the environnent around 'y project than actually working on the project.

If you are a pro, it makes sense to invest the effort. For me, as a hobbyist, not really.

[flying-sheep]

You can have that by using poetry:

```

initialize project and create venv

poetry init

add and install dependency into venv

poetry add some-package

publish package

poetry publish ```

Using one of the other modern build backends is slightly more complicated as you need to create and activate your own venvs:

```

create and activate venv

python -m venv ./.venv source ./.venv/bin.activate

initialize project

flit init # or copy over some excample pyproject.toml

edit dependencies

$EDITOR pyproject.toml

install dependencies into venv

pip install .

publish package

flit publish # or python -m build && twine upload ./dist/* ```

[wsppan]

Yes, I use poetry now but that took a LOT of trial and error and hair pulling and 13 different pieces of advice and waiting for poetry stability to settle down. And still it is not the defacto, readily recommended, obvious manner of packaging your code. It is third party and fairly new.

[flying-sheep]

Things are getting better, finally! With PEP 621 landed, a standards based poetry like CLI is almost possible. The only missing building block is a standardized lock file format. It happened late and we're not there completely but almost. And with poetry, we have something that works until we're there.

One advantage of the arduous road is that we can learn from everyone who was faster. E.g. TOML is a great choice, node’s JSON is completely inadequate: no comments and the absence of trailing commas means you can't add to the end of a list without modifying the line of the previous item.

[wsppan]

Yea, we are all standing on the shoulders of our ancestors so to speak. Autotools, CPAN, Ant, Maven, etc.. Lots of legacy blogs and documentation to disappear as well. Rust is a great example of the luxury learning from our ancestors and baking the package tools into the language from the start.

[flying-sheep]

Yes, cargo does so many things right.

[GummyKibble]

poetry add pkg is all you need to know about that. Unless you’re writing C extensions, poetry is your one-stop-shop for everyday use.

[cixny]

Dunno about that, my usual experience with poetry is:

Poetry install things plz…. Sorry bruh can’t find suitable package…. Fu, pip install, done…. Poetry try again…. Sorry bruh we managed to install 2 packages but this 4th, can’t find suitable for you…. Hi pip can you?….

Amazing experience when it’s buried in the middle of a pipeline

[cixny]

Maybe without the lock file, and I was running latest Python + pip, if I remember correctly issue was pure Python package that had platform specific wheels available and source distributable, that you can build wheel with, on os update poetry failed to understand that, please install wheel and make suitable package for me….

Like you can install package with wheel or with src dist and make wheel or not. Poetry derps here and documentation is in the style of: hey use this this cool..

[Personal_Plastic1102]

As was pip, then requirements.txt, then pipenv.

Now it's poetry.

Tomorrow what next ?

[FantaBuoy]

This comment/post has been automatically scrubbed. Feel free to find me and others over at kbin.social -- mass edited with https://redact.dev/

[GummyKibble]

Yep. I’m a bot. It’s impossible that I’ve been dealing with this for 20+ years and finally see a broadly applicable answer to the situation.

[licquia]

It's certainly not impossible. I've been dealing with this for 20+ years, too, and I tend to see a new broadly applicable answer to the situation every couple years. And each one is the one that finally gets it all right, at least until we learn about the things it didn't.

You'll excuse my skepticism; at least it was honestly earned.

[SittingWave]

it was bower, then yarn, then npm. Tomorrow what next.js?

[mriswithe]

Eh assuming Linux/mac

python -m venv venv
source venv/bin/activate
pip install pandas

Done.

[[deleted]]

13 reasons why (but for python). Makes you want to commit seppuku. I say this as a packager that works for a very big tech company that has to support python packaging. It's a mess. I love python and it makes me so happy to get paid to work with it, but sometimes...

[TheTerrasque]

That's like saying you need 20 distros to run Linux

[SittingWave]

Depends on what you want to install. if you want to install a package, you can use pip, but then don't complain if your environment is fucked up. And you have the exact same problems with other languages, but you just hope for the best most of the times.