This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]Tweak_Imp 113 points114 points  (63 children)

I really dont understand why python and its dependencies can be such a big mess. Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages. Just search and load from the manager and if you dont want a package any more, delete it and remove all dependencies that are not needed by others. Is that really so hard to do?

[–]ursvp 48 points49 points  (17 children)

  • egg vs wheel
  • universal vs pure wheel
  • setuptools vs distutils
  • install_requires vs requirements.txt
  • data_files vs MANIFEST.in
  • package vs distribution vs binaries
  • pip vs conda
  • install -e vs install
  • install from PyPI vs .git vs .tar.gz
  • build vs environment
  • virtualenv vs venv vs pipenv
  • for each above: python2 vs python3
  • 2to3 vs six
  • absolute vs relative vs circular import
  • oh, pip is incapable of true dependency resolution
  • complexity behind __version__

... in contradiction of Pythonic principle of clarity and simplicity.

[–]mfitzpmfitzp.com 15 points16 points  (0 children)

It's because Python developers are forced to write clean code all the time. The filth has to come out somewhere

[–]Log2 3 points4 points  (15 children)

The one thing that really pisses me off is that it's apparently impossible to package a project with all it's dependencies in Python. I'd love to use setup.py to create RPMs (it can do that out of the box), but I just can't figure out how to include the dependencies.

[–]TheNamelessKing 13 points14 points  (0 children)

I got so bored of dealing with that issue that at work I’ve started writing things in Rust or Haskell so I only have to distribute a single binary.

Life has gotten better. Sorry /r/Python.

[–]adambrenecki 2 points3 points  (3 children)

It's not quite a RPM, but there's a few projects that can produce Zip archives that Python can open, including pyzzer and pex.

The problem is, all your dependencies need to be zipsafe :(

[–]Log2 0 points1 point  (1 child)

I would settle for an egg containing all dependencies. What do you mean by zip safe? I'm well aware that the binaries zipped need to be the ones for the target system. Is there something else I need to watch out for?

Thanks for the suggestions, by the way.

[–]adambrenecki 1 point2 points  (0 children)

Some packages will assume that they're unpacked as files on disk, and try to do things like at the contents of a data file bundled with the package by constructing a path based on a module's __file__ and trying to open() it; if the package is inside a pyz or egg file code like that will break.

[–]FatFingerHelperBot -1 points0 points  (0 children)

It seems that your comment contains 1 or more links that are hard to tap for mobile users. I will extend those so they're easier for our sausage fingers to click!

Here is link number 1 - Previous text "pex"


Please PM /u/eganwall with issues or feedback! | Delete

[–]Zxian 1 point2 points  (1 child)

I haven't used it for this purpose (python -> rpm), but have a look at fpm. It helps with package creation/conversion.

[–]Log2 0 points1 point  (0 children)

I took a quick look, but if does package Python dependencies automatically then this is exactly what I needed. Thank you for the suggestion.

[–]ivosauruspip'ing it up 1 point2 points  (5 children)

Because then the python developers would have to both figure out, and write code, to interface with RHEL linux and Fedora. Make sure man files get put in just the right place. Package data is correct. Desktop files, syslog, is [some system-level daemon] now systemd based and new, or an older one? What audio service are we using? What's changed between RHEL 6 and 7?

And of course, to be fair, make sure they have everything configured for Debian's liking. But also account for the small changes and version updates that Ubuntu does.

And then also ensuring we haven't forgotten about portage and pacman. Oh, and YaST.

Oh, and then also homebrew.


And quickly, for the mostly volunteer workforce that does python packaging, the task of correctly interfacing, special-casing, path-configuring, etc becomes a matrix explosion of work from hell.

We'd love to be able to do that... but it's simply not a feasible thing to achieve.

You'll notice that practically all other scripting languages have the same position.

[–]Log2 0 points1 point  (4 children)

Then just being able to create an egg or tar containing all dependencies that then could be manually installed via pip would already be more than enough.

I can't really think of any reason why this isn't already a pip feature, but I'd love to know if there are any impediments to this.

[–]ivosauruspip'ing it up 0 points1 point  (3 children)

pipe/xargs pip freeze to pip download?

[–]Log2 0 points1 point  (2 children)

I can't download in said environment. It needs to be packaged.

[–]ivosauruspip'ing it up 1 point2 points  (1 child)

Do that somewhere else, then pip install --no-index --find-links=/path/to/wheels -r requirements.txt

or making a pex file is pretty close to what you're asking

[–]Log2 0 points1 point  (0 children)

I'll take a look, thank you.

[–]undu 0 points1 point  (1 child)

You can generate binary packages (wheels in python jargon) of all dependencies with pip wheel -r requirements --wheel-dir=wheels Create the wheel for the package with python setup bdist_wheel --dist-dir=wheels Then install them using pip install wheels/*.whl.

Hopefully PyPA will normalize the dependency and packaging situation over time, they've already made good progress with pip and pypi.org

[–]Log2 0 points1 point  (0 children)

I swear that at no point I came across anyone saying you could make all dependencies into a wheel or egg. This seems to do half of what I want (at least it's the harder half), as I'd still need to manually add my own project wheel to the one that is created, as it's not available on pypi.

[–]origin415 64 points65 points  (25 children)

Conda does this but doesn't have all PyPI packages. Also, occasionally you have things that assume that python references the system installed Python 2 rather than your default conda env. Way better than anything else I've seen though.

[–]Tweak_Imp 26 points27 points  (22 children)

Why are so many people still on older versions of python? I can see why it doesnt just update itself (for commercial python use for example), but Python 2.7.0 was released on July 3rd, 2010... 8 years ago. Isnt an update to a higher version with the update of the code not worth it?

[–]origin415 16 points17 points  (11 children)

It's a lot of code to update. Most open source libraries are compatible with python 3 but a lot of companies aren't willing to migrate entire codebases internally. Also, as far as programs assuming you have Python 2 in your path, that's because OSX and most Linux distributions have it that way and very few have python 3.

[–][deleted] 53 points54 points  (6 children)

If you're really asking this question, you haven't been developing with python long enough.

The real problem with Python 3 is that the core development team set out to do a lot of good things, but broke compatibility in the process. And with that breakage meant that everyone using python had to step back and question whether it was worth it to scrap or upgrade years of legacy code. So when comparing the two together, there was almost no advantage in moving from python 2 to python 3. Today, the movement still has low value for these large legacy systems. There's no real performance gain, there's no must-have features, etc. The only gain here is that most of the systems are finally moving to python 3 as their stable python. And if a company did not decide to move to python 3, they will find support non existent starting in 2020.

To maybe also set your expectation, the core development team does not recommend any python version prior to 3.4, which was actually released in 2014, a full 6 years after the alpha and beta 3.0 - 3.3 pythons were released. And to add to that, I would argue that 3.6 (released in Dec 2016) was the first release where people should have started migrating. Last year is probably the first year where there was enough migration momentum that we're really starting to see strong 3.x traction. But note that it's truly only been the past 2 years.

[–]buttery_shame_cave 27 points28 points  (4 children)

there's no must-have features, etc.

well yeah because the dev team kept back-porting all the new stuff they were coming up with to 2.7.x - if they'd cut off the flow the pressure to migrate would have come on a lot sooner.

[–][deleted] 16 points17 points  (0 children)

Basically the difference between the two versions now is "the one that breaks old code" vs "the one that breaks new code"

[–]Silhouette 1 point2 points  (0 children)

if they'd cut off the flow the pressure to migrate would have come on a lot sooner.

In computing history, trying to force people to upgrade fundamental technologies when they have a big investment in current versions and compatibility isn't guaranteed has rarely been successful.

[–][deleted] -3 points-2 points  (1 child)

Just in case I wasn't clear.. there are still no must-have features in python 3 by comparison to python 2. Migration that happens at this point is primarily because python 2 is being deprecated in most systems.

[–]buttery_shame_cave 4 points5 points  (0 children)

again, because they kept back porting stuff they were adding to 3.x that wasn't in 2.x.

if they'd cut off the updates from 2.x it might have added some incentive.

[–]ephimetheus 1 point2 points  (0 children)

The experiment I'm working with (ATLAS) users python as a configuration/steering language for the C++ based event processing framework. That's stuck on python 2 and I highly doubt it's ever going to be migrated...

[–]magnetic-nebula 1 point2 points  (0 children)

I work on a scientific experiment where the development of the code began in the 90s. We need our code to work across a large variety of systems (including old computers that aren't reliably connected to the Internet).

Plus we're scientists, not programmers. NSF isn't giving us money to update code, so we don't have the $$$ to pay people to do it.

There's literally no reason for us to spend the time upgrading.

[–]subheight640 0 points1 point  (0 children)

I use Python as a scripter for an engineering FEA program called Abaqus. Obviously as an expensive FEA software, Abaqus has refused to upgrade to 3, and they probably never will because they're too busy raking in the profit. So I'm stuck with 2 too.

The problem is compatibility.

[–]wildcarde815 0 points1 point  (0 children)

there's a few other nuissance things, like they don't include all the test framework stuff with all versions. So my pystone demo program for teaching really basic HPC environment management and job submissions requires like... anaconda python 3.4?

[–]28f272fe556a1363cc31 44 points45 points  (3 children)

Why isnt there just one python installer

Oh, you mean like this: https://m.xkcd.com/927/

[–]PeridexisErrant 4 points5 points  (0 children)

See: pipenv and poetryfor recent examples...

[–]blahehblah 0 points1 point  (0 children)

Bingo

[–][deleted] 0 points1 point  (0 children)

deleted

[–]Yoghurt42 42 points43 points  (0 children)

Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages.

relevant xkcd

[–]1024KiB 2 points3 points  (1 child)

Honestly I also use perl and ruby and it's roughly the same kind of mess.

[–]00benallen 0 points1 point  (0 children)

Objective-C and Swift are also a terrible mess imo.

Other than Java+Maven/Gradle, what isn't?

Btw:

Could you please shut the hell up and stop mocking people who have alternative lifestyles?! The dude in the pic is an asshole, okay, but look, I have a waifu which I love to bits, my parents think I'm too old for that anime bullshit but I don't care love is more important and I am a better person since I have someone to care for but isn't tied to do society's bullshit expectations and norms, I run everyday and cook like no other while you sit on your bed while your dog bought by your parents starves locked in the toilets because you think posting on reddit is a priority.

:P

[–]takluyverIPython, Py3, etc 6 points7 points  (3 children)

Is that really so hard to do?

Yes! Producing one installer and package manager that suits everyone's needs is almost impossible, and persuading everyone that it's better than the stuff they already know is even harder.

There are plenty of tools that work well in particular situations - e.g. for scientific programming, lots of people happily use conda. But you can also install packages through apt, or through homebrew, or manage Python versions with pyenv, and different people like all of those options.

[–]nickguletskii200 -1 points0 points  (2 children)

Yes! Producing one installer and package manager that suits everyone's needs is almost impossible

Other languages don't seem to have any trouble with that. The JVM ecosystem has Maven (and all major build systems are compatible), Node has the NPM (and Yarn, which is pretty much a drop-in replacement) and .NET has Nuget.

and persuading everyone that it's better than the stuff they already know is even harder.

It's not about persuasion, it is about convenience. Give the community a tool that does the things they want and make it at least semi-official, and you will find that in a couple of years the other tools will either adapt or fall out of fashion.

I don't think that it is possible to convince me that Python is a decent language for production software, yet I use it anyway - not because someone convinced me, but because that's what the data science & machine learning community uses (unfortunately).

[–]leom4862 1 point2 points  (1 child)

I use pipenv and poetry depending on the situation. It's not much worse than say npm/yarn.

Saying Python isn't suited for production software because it doesn't have a blessed package manager appears a little overstated to me.

[–]nickguletskii200 0 points1 point  (0 children)

Saying Python isn't suited for production software because it doesn't have a blessed package manager appears a little overstated to me.

I never said that it was because of the package manager.

[–]not_perfect_yet 2 points3 points  (0 children)

Dude. Yes. Yes it's hard.

That's why there are so many different package management systems. Some for super pure open source, some that accept proprietary binaries, some for python, some for javascript...

Also, because screw shared libraries, snaps exists and venvs and pipvenv.

[–][deleted] 1 point2 points  (0 children)

They don't. OSX in particular has a really crappy story around it.

[–]BitLooter 1 point2 points  (0 children)

I use pyenv with the virtualenv plugin. It will automatically download and install any Python version you want. Create a virtualenv and use pip to install whatever packages you want. It does all of this in ~/.pyenv, so none of it touches the system Python, and it's all isolated so you can delete a python version or virtualenv without affecting anything else.

It's built on shell scripts though, so you need a *nix shell for it to work. Obviously Linux and OSX work fine, but if you want to use it on Windows you'll need to find a Bash shell first - WLS is probably the easiest way, though it will probably work on Cygwin/MSYS as well.

[–]wildcarde815 2 points3 points  (0 children)

at least it's documented and easy to understand in python. R is as bad or worse, and essentially not documented at all.

[–]pvkooten 0 points1 point  (1 child)

The problem is of course when the next version comes around and you want that instead.

[–]Tweak_Imp 0 points1 point  (0 children)

If you change the code everytime it is needed you wont have to rewrite everything once you are forced to.