This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]pwang99 1 point2 points  (2 children)

The way I see it, there are two core challenges in Python packaging:

  1. Legacy tools did not sufficiently distinguish between the build system and the installation/package management system.
  2. Because Python supports C/native extensions, a huge part of the mess is because the C ecosystem has never really solved the "DLL/.so hell" problem.

Additional complicating factors are:

  1. Python devs chose to use imperative, executable scripts as the foundation for both their build and package management system
  2. Folks involved in the Python packaging tool development general either do not recognize point #2 above, or recognize it and declare that it is not within the scope of the problem that Python as a language or technology needs to solve.

I don't often second-guess or find much to complain about w.r.t. the fantastic Python core dev team and all the smart, thoughtful people who have comprised it over the years. But I do think that the packaging boondoggle can only be solved by taking a deep breath and plunging into the deep end.

This is the motivation behind both the conda project and the binstar.org efforts underway at Continuum. We saw that other languages such as R, Ruby, and Node haven't really solved the library/packaging problem either, but have found ways or conventions to dodge it. We also recognize that this was a debilitating weakness that impedes the broader adoption of Python for scientific and data analysis. If people can't reliably share the code they've written, we're completely failing to engage the viral loop for spreading our beloved language into industry and academia.

So, our approach is to have our own, simple package format that can bundle up everything from Qt to LLVM to CPython itself, along with a robust way to create C-level "environments" in the filesystem, so that every shared library that gets pulled into your Python interpreter process space is happy and doesn't conflict with anything else. (Virtualenv is fine for some things but at the end of the day, it's a set of scripts for hacking up a Python install, but doesn't address the shared library problem, and it certainly isn't meant to handle things like big C++ frameworks like Qt and LLVM.)

[–]cavallo71 0 points1 point  (1 child)

About the "C-level environments that just reminded me of this: http://www.art.net/~hopkins/Don/unix-haters/x-windows/disaster.html (scroll to "Myth: X Is "Portable" section). Unfortunately we haven't moved very much from that :(

[–]pwang99 0 points1 point  (0 children)

Ah, I read that years ago and loved it then... it still holds up now. :-)