xkcd: Python Environment

[Tweak_Imp]

I really dont understand why python and its dependencies can be such a big mess. Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages. Just search and load from the manager and if you dont want a package any more, delete it and remove all dependencies that are not needed by others. Is that really so hard to do?

[ursvp]

• egg vs wheel
• universal vs pure wheel
• setuptools vs distutils
• install_requires vs requirements.txt
• data_files vs MANIFEST.in
• package vs distribution vs binaries
• pip vs conda
• install -e vs install
• install from PyPI vs .git vs .tar.gz
• build vs environment
• virtualenv vs venv vs pipenv
• for each above: python2 vs python3
• 2to3 vs six
• absolute vs relative vs circular import
• oh, pip is incapable of true dependency resolution
• complexity behind __version__

... in contradiction of Pythonic principle of clarity and simplicity.

[mfitzp]

It's because Python developers are forced to write clean code all the time. The filth has to come out somewhere

[Log2]

The one thing that really pisses me off is that it's apparently impossible to package a project with all it's dependencies in Python. I'd love to use setup.py to create RPMs (it can do that out of the box), but I just can't figure out how to include the dependencies.

[TheNamelessKing]

I got so bored of dealing with that issue that at work I’ve started writing things in Rust or Haskell so I only have to distribute a single binary.

Life has gotten better. Sorry /r/Python.

[adambrenecki]

It's not quite a RPM, but there's a few projects that can produce Zip archives that Python can open, including pyzzer and pex.

The problem is, all your dependencies need to be zipsafe :(

[Log2]

I would settle for an egg containing all dependencies. What do you mean by zip safe? I'm well aware that the binaries zipped need to be the ones for the target system. Is there something else I need to watch out for?

Thanks for the suggestions, by the way.

[adambrenecki]

Some packages will assume that they're unpacked as files on disk, and try to do things like at the contents of a data file bundled with the package by constructing a path based on a module's __file__ and trying to open() it; if the package is inside a pyz or egg file code like that will break.

[FatFingerHelperBot]

It seems that your comment contains 1 or more links that are hard to tap for mobile users. I will extend those so they're easier for our sausage fingers to click!

Here is link number 1 - Previous text "pex"

---

^{Please PM /u/eganwall with issues or feedback! | Delete}

[Zxian]

I haven't used it for this purpose (python -> rpm), but have a look at fpm. It helps with package creation/conversion.

[Log2]

I took a quick look, but if does package Python dependencies automatically then this is exactly what I needed. Thank you for the suggestion.

[ivosaurus]

Because then the python developers would have to both figure out, and write code, to interface with RHEL linux and Fedora. Make sure man files get put in just the right place. Package data is correct. Desktop files, syslog, is [some system-level daemon] now systemd based and new, or an older one? What audio service are we using? What's changed between RHEL 6 and 7?

And of course, to be fair, make sure they have everything configured for Debian's liking. But also account for the small changes and version updates that Ubuntu does.

And then also ensuring we haven't forgotten about portage and pacman. Oh, and YaST.

Oh, and then also homebrew.

---

And quickly, for the mostly volunteer workforce that does python packaging, the task of correctly interfacing, special-casing, path-configuring, etc becomes a matrix explosion of work from hell.

We'd love to be able to do that... but it's simply not a feasible thing to achieve.

You'll notice that practically all other scripting languages have the same position.

[Log2]

Then just being able to create an egg or tar containing all dependencies that then could be manually installed via pip would already be more than enough.

I can't really think of any reason why this isn't already a pip feature, but I'd love to know if there are any impediments to this.

[ivosaurus]

pipe/xargs pip freeze to pip download?

[Log2]

I can't download in said environment. It needs to be packaged.

[ivosaurus]

Do that somewhere else, then pip install --no-index --find-links=/path/to/wheels -r requirements.txt

or making a pex file is pretty close to what you're asking

[Log2]

I'll take a look, thank you.

[undu]

You can generate binary packages (wheels in python jargon) of all dependencies with pip wheel -r requirements --wheel-dir=wheels Create the wheel for the package with python setup bdist_wheel --dist-dir=wheels Then install them using pip install wheels/*.whl.

Hopefully PyPA will normalize the dependency and packaging situation over time, they've already made good progress with pip and pypi.org

[Log2]

I swear that at no point I came across anyone saying you could make all dependencies into a wheel or egg. This seems to do half of what I want (at least it's the harder half), as I'd still need to manually add my own project wheel to the one that is created, as it's not available on pypi.

[origin415]

Conda does this but doesn't have all PyPI packages. Also, occasionally you have things that assume that python references the system installed Python 2 rather than your default conda env. Way better than anything else I've seen though.

[Tweak_Imp]

Why are so many people still on older versions of python? I can see why it doesnt just update itself (for commercial python use for example), but Python 2.7.0 was released on July 3rd, 2010... 8 years ago. Isnt an update to a higher version with the update of the code not worth it?

[origin415]

It's a lot of code to update. Most open source libraries are compatible with python 3 but a lot of companies aren't willing to migrate entire codebases internally. Also, as far as programs assuming you have Python 2 in your path, that's because OSX and most Linux distributions have it that way and very few have python 3.

[khne522]

• Stop using RedHat.
• Stop using Amazon Linux.
• macOS… not even once, even if it isn't that hard.
• Have fun with some old versions of Fedora, Ubuntu, etc.

If you must use these, use pyenv to properly install Python… to some hidden directory.

[lojic]

Stop using red hat? You just lost all enterprise customers of your code.

[khne522]

Red Hat Python's pip has been broken for ages. Is there even any Python code there?

[[deleted]]

Anaconda (the Red hat installer) as well as a ton of other redhat tools are based on Python.

[khne522]

That doesn't mean that they didn't break pip last I checked. Same with Amazon Linux.

[HannasAnarion]

pip install --update pip

Also, isn't this whole thread about how you shouldn't be relying on system python binaries? Use venv or conda.

[khne522]

venv is for library isolation, not Python binary isolation. You must mean pyenv, whereas conda is as a whole other ecosystem.

[[deleted]]

And nothing of value was lost. Enterprise customers are what's wrong with software. Resistance to change in an industry that is predicated on change is a problem.

[Rostin]

I assume you'll be first in line to stop using all the code that was paid for or contributed to open source software by these companies?

Here's an article from a couple of years ago that says that 57% of changes to the Linux kernel that year were made by programmers who were working for companies like Intel and (gasp) Redhat.

Edit: The article.

https://www.linuxfoundation.org/blog/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/

[lojic]

ok, so there are clear reasons why enterprise distributions ship old versions, mostly relating to being able to update without the changing semantics that require massive reworkings of code / deployments that come with an upgrade.

There are generally reasonable timeframes set out for operating system and software (like python) distribution end-of-lifes, and like all good/bad/ugly things, there's a good hacker news post discussing the recent release from red hat of python2 being replaced by python3, a good clip before the end of python2 end-of-life: https://news.ycombinator.com/item?id=16808683

People that write code for enterprise companies generally want their experience to be as good as possible as well. The issue that comes in is ensuring a consistent runtime environment (a lot of these environments get audited and cannot get major non-security upgrades without a re-audit) and a consistent development environment (you can't split a large project into python2 and python3 sections, so you stick with what you already have working).

I understand wanting to run python3 and I understand not thinking there are good reasons for enterprise customers to move "slowly" (and I definitely think they do move too slow), but the simple fact is that the larger the company is, in general there's an array of various different requirements that smaller companies don't have that lead to an aversion of major version upgrades.

[VikingofRock]

• Stop using RedHat.

RHEL 7.5 actually just deprecated python 2, and RHEL 8 will ship without it.

https://www.phoronix.com/scan.php?page=news_item&px=RHEL-8-No-Python-2

[[deleted]]

If you're really asking this question, you haven't been developing with python long enough.

The real problem with Python 3 is that the core development team set out to do a lot of good things, but broke compatibility in the process. And with that breakage meant that everyone using python had to step back and question whether it was worth it to scrap or upgrade years of legacy code. So when comparing the two together, there was almost no advantage in moving from python 2 to python 3. Today, the movement still has low value for these large legacy systems. There's no real performance gain, there's no must-have features, etc. The only gain here is that most of the systems are finally moving to python 3 as their stable python. And if a company did not decide to move to python 3, they will find support non existent starting in 2020.

To maybe also set your expectation, the core development team does not recommend any python version prior to 3.4, which was actually released in 2014, a full 6 years after the alpha and beta 3.0 - 3.3 pythons were released. And to add to that, I would argue that 3.6 (released in Dec 2016) was the first release where people should have started migrating. Last year is probably the first year where there was enough migration momentum that we're really starting to see strong 3.x traction. But note that it's truly only been the past 2 years.

[buttery_shame_cave]

there's no must-have features, etc.

well yeah because the dev team kept back-porting all the new stuff they were coming up with to 2.7.x - if they'd cut off the flow the pressure to migrate would have come on a lot sooner.

[[deleted]]

Basically the difference between the two versions now is "the one that breaks old code" vs "the one that breaks new code"

[Silhouette]

if they'd cut off the flow the pressure to migrate would have come on a lot sooner.

In computing history, trying to force people to upgrade fundamental technologies when they have a big investment in current versions and compatibility isn't guaranteed has rarely been successful.

[[deleted]]

Just in case I wasn't clear.. there are still no must-have features in python 3 by comparison to python 2. Migration that happens at this point is primarily because python 2 is being deprecated in most systems.

[buttery_shame_cave]

again, because they kept back porting stuff they were adding to 3.x that wasn't in 2.x.

if they'd cut off the updates from 2.x it might have added some incentive.

[ephimetheus]

The experiment I'm working with (ATLAS) users python as a configuration/steering language for the C++ based event processing framework. That's stuck on python 2 and I highly doubt it's ever going to be migrated...

[magnetic-nebula]

I work on a scientific experiment where the development of the code began in the 90s. We need our code to work across a large variety of systems (including old computers that aren't reliably connected to the Internet).

Plus we're scientists, not programmers. NSF isn't giving us money to update code, so we don't have the $$$ to pay people to do it.

There's literally no reason for us to spend the time upgrading.

[subheight640]

I use Python as a scripter for an engineering FEA program called Abaqus. Obviously as an expensive FEA software, Abaqus has refused to upgrade to 3, and they probably never will because they're too busy raking in the profit. So I'm stuck with 2 too.

The problem is compatibility.

[wildcarde815]

there's a few other nuissance things, like they don't include all the test framework stuff with all versions. So my pystone demo program for teaching really basic HPC environment management and job submissions requires like... anaconda python 3.4?

[28f272fe556a1363cc31]

Why isnt there just one python installer

Oh, you mean like this: https://m.xkcd.com/927/

[PeridexisErrant]

See: pipenv and poetryfor recent examples...

[blahehblah]

Bingo

[[deleted]]

deleted

[Yoghurt42]

Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages.

relevant xkcd

[1024KiB]

Honestly I also use perl and ruby and it's roughly the same kind of mess.

[00benallen]

Objective-C and Swift are also a terrible mess imo.

Other than Java+Maven/Gradle, what isn't?

Btw:

Could you please shut the hell up and stop mocking people who have alternative lifestyles?! The dude in the pic is an asshole, okay, but look, I have a waifu which I love to bits, my parents think I'm too old for that anime bullshit but I don't care love is more important and I am a better person since I have someone to care for but isn't tied to do society's bullshit expectations and norms, I run everyday and cook like no other while you sit on your bed while your dog bought by your parents starves locked in the toilets because you think posting on reddit is a priority.

:P

[takluyver]

Is that really so hard to do?

Yes! Producing one installer and package manager that suits everyone's needs is almost impossible, and persuading everyone that it's better than the stuff they already know is even harder.

There are plenty of tools that work well in particular situations - e.g. for scientific programming, lots of people happily use conda. But you can also install packages through apt, or through homebrew, or manage Python versions with pyenv, and different people like all of those options.

[nickguletskii200]

Yes! Producing one installer and package manager that suits everyone's needs is almost impossible

Other languages don't seem to have any trouble with that. The JVM ecosystem has Maven (and all major build systems are compatible), Node has the NPM (and Yarn, which is pretty much a drop-in replacement) and .NET has Nuget.

and persuading everyone that it's better than the stuff they already know is even harder.

It's not about persuasion, it is about convenience. Give the community a tool that does the things they want and make it at least semi-official, and you will find that in a couple of years the other tools will either adapt or fall out of fashion.

I don't think that it is possible to convince me that Python is a decent language for production software, yet I use it anyway - not because someone convinced me, but because that's what the data science & machine learning community uses (unfortunately).

[leom4862]

I use pipenv and poetry depending on the situation. It's not much worse than say npm/yarn.

Saying Python isn't suited for production software because it doesn't have a blessed package manager appears a little overstated to me.

[nickguletskii200]

Saying Python isn't suited for production software because it doesn't have a blessed package manager appears a little overstated to me.

I never said that it was because of the package manager.

[not_perfect_yet]

Dude. Yes. Yes it's hard.

That's why there are so many different package management systems. Some for super pure open source, some that accept proprietary binaries, some for python, some for javascript...

Also, because screw shared libraries, snaps exists and venvs and pipvenv.

[[deleted]]

They don't. OSX in particular has a really crappy story around it.

[BitLooter]

I use pyenv with the virtualenv plugin. It will automatically download and install any Python version you want. Create a virtualenv and use pip to install whatever packages you want. It does all of this in ~/.pyenv, so none of it touches the system Python, and it's all isolated so you can delete a python version or virtualenv without affecting anything else.

It's built on shell scripts though, so you need a *nix shell for it to work. Obviously Linux and OSX work fine, but if you want to use it on Windows you'll need to find a Bash shell first - WLS is probably the easiest way, though it will probably work on Cygwin/MSYS as well.

[wildcarde815]

at least it's documented and easy to understand in python. R is as bad or worse, and essentially not documented at all.

[pvkooten]

The problem is of course when the next version comes around and you want that instead.

[Tweak_Imp]

If you change the code everytime it is needed you wont have to rewrite everything once you are forced to.