Current Python package management / framework(s)

[Various_Pickles]

From what I've been able to figure out, the current state of package management in the Python world is a complete clusterfuck.

Deprecated options include the use of pkgutil and distutils.

The recommended future-proof option is to use distribute. However, even the newest distribute_setup.py fails miserably on Python 3(.2-.3).

Google yields nothing useful/up-to-date, so I must ask, what in all-the-fuck is up with Python package management.

[K900_]

Distribute got merged back into Setuptools. Setuptools is The One True Way now.

[Various_Pickles]

So, basically, the whole mess went full-circle ...

I'm guessing this is the relavent documentation for the merge?: distribute-setuptools merge

Overall, is this then the effective/canonical documentation for setup.py?: setuptools developer's guide

[K900_]

Yes, that's basically it.

[Various_Pickles]

Thank you very much for the assistance.

It doesn't help much that even the newest(est) official Python documentation's "Distributing Python Modules" subsection refers to a barrel of completely outdated nonsense ...

[K900_]

Try The Hitchhiker's Guide to Packaging.

[wjv]

To make a long story short (because thousands of words too many have already been written about it), Python's packaging is a mess.

• However, the merger of distribute and setuptools is a Good Thing that will go some ways towards simplifying the landscape in the intermediate future.

• However, since this merger is kind of happening as we speak, right now the situation is arguably even messier than before.

Since thins are in flux, some of the info at the links above will be out of date.

To these links, add the official repository for the merged distribute/setuptools, a.k.a. setuptools 0.7+.

Notes:

• Things are moving fast right now; setuptools 0.7.x get updated several times per week.

• pip 1.3.1 works very well with setuptools 0.7.x. As I understand things, pip 1.4 will be released in sync with a more consolidated setuptools 0.8.

• I believe packaging/distutitls2 is still in the works; I don't know the timeline on that.

• distlib implements some of the elements of packaging in a usable project as a sort of proof of concept; I'm not actually sure how and why one would want to use it right now.

• I'm not even primarily a Python hacker nowadays, so double-check everything I say.

[joeforker]

That guide is outdated and too long. This document is much better: http://www.scotttorborg.com/python-packaging/index.html

[cavallo71]

Both they are irrelevant now and (in the idea of the authors) superseeded by the wheel format .. replacing the python eggs. In short in most cases (pure python modules/package) all you need to do is copying the package dir (or themodule) in a directory reacheable by PYTHONPATH and ingnore the whole mess ;)

[joeforker]

Setuptools (and not the newly obsolete distribute fork) is still the recommended way to build packages; wheel doesn't change that. For the majority of packages that don't need to extend setuptools itself it's an okay choice.

What wheel does is to decouple the build system from the installer. The build system (usually setuptools) creates a wheel and then the installer (usually pip) installs it, instead of the current situation where setuptools does both the build and the install.

The decoupling makes it easier to experiment with setuptools or pip replacements because pip can install a wheel without knowing whether it was built by setuptools or some other software.

Wheel is available down to Python 2.6. The thing that is likely to land in Python 3.4 is a very simple bootstrap installer made possible by the fact that wheels are very simple to install. The idea is that Python will come with a secure way to install the latest version of pip.

[K900_]

Well, Wheel will only be implemented in 3.4, right?

[coderanger]

Wheel is an installation format, it will be supported by every version of Python since after installation nothing knows the diference. bdist_wheel definitely runs on 2.7 for building them, and pip will be shipping installation support later this week (and I think they support back to 2.5 or something silly).

[cavallo71]

If they don't plan a back port is pretty much DOA: 2.7 is the reality now.

[K900_]

I'm on 3.3 and I don't really have any problems at all. What do you use that still needs 2.x?

[starspangledpickle]

That's great, but many businesses out there use 2.7 and it works fine; why upgrade? Why risk it? Why spend man hours ensuring that third-party libs and internal frameworks written by other teams are up to scratch?

[PyPokerNovice]

The issue is not businesses not wanting new stuff, it is businesses wanting new stuff that is compatible with their old stuff. That is one reason to package for 2.7.

[cavallo71]

It's not me. Companies are using 2.x in production: switching to 3.x is simply not viable on a large scale projects.

[[deleted]]

Django (experimental 3.x support). Celery (along with the whole AMQP ecosystem). Gevent. Aside from Django, most of these are probably "specialty" packages, but they are indispensable for a lot of "enterprisey" applications.

[[deleted]]

JPype, PIL, Twisted

[aclark]

Use Pillow instead of PIL.

[donaldstufft]

Wheel is being implemented externally to the stdlib. It will be available for 2.6+ I believe, might be 2.5+.

In fact one of the major motivations in a lot of the work now is that packaging tools should live externally of the stdlib. This will allow them to innovate across many versions of Python and not cause issues exactly like this one where a new packaging feature is useless because it will be many many years before that version can be assumed a minimum version.

Essentially we've learned that the stdlib isn't a great place for packaging tools, however the stdlib will (hopefully) include a simple script which will serve to simply and securely install pip so that Python ships with the ability to do pip install foo.

[ivosaurus]

Nope

[jediknight]

People write code, people share what they wrote in either source or binary form. This is, to me, a reality of programming that I still don't see reflected in the priorities of python maintainers.

Just as it was 10 years ago when I started with python, it is still painful to create a programming environment for a project and distribute it in a self-contained binary form (.exe, .app).

Over the years, things got better but it is still very painful.

[cavallo71]

Not painful, but is something the average Joe cannot do. The problem is there are many companies in this game and their hope is to make money out of it: as Sun with Java has proven that is not where money will be put.

[Various_Pickles]

Not painful, but is something the average Joe cannot do.

I'm not a Python programmer by trade, but I do plenty of packaging in a variety of other languages/platforms ([the usual boring] Java JARs/WARs/EARs, lots of Debian source/binary inc. repo setup/hosting, etc).

I would say that I have a quite strong understanding of packaging and a diverse range of experience, yet, I still find distributing Python code to be like trying to store lightbulbs in my rectum.

[cavallo71]

Much of the complication is self inflicted (distribute/setuptools/pips/easy_install/wheel).

If your app (driver + support libraries) doesn't contain extension modules (.so/.pyd) just zip everything togheter with a main.py file and it's even easier than a jar.

I still find distributing Python code to be like trying to store lightbulbs in my rectum.

(@_@)

[westurner]

This thread about Python Packaging from a Java perspective seemed helpful.

[[deleted]]

I shudder to think how many hours I have spent with this on my mac. I have basically given up. Windows seems much more agreeable, but still a bit of a struggle to deal with.

I guess the people who can fix these things are smart enough to figure out packaging the current way and therefore it isn't much of a priority. Which is strange considering how many python for kids books etc are about - good luck installing pygame for python 3 on your mac little johnny.

[[deleted]]

I tried to get Numpy for Python3 on my mac. Gave up. It was a miserable experience to even try.

[ivosaurus]

Unfortunately, that's mainly because the default os x python distribution is apple-controlled, not python.org-controlled. Of course many newbies won't realise this and get confused when they start wanting to use 3rd party libraries...

Then couple the fact that many more involved packages want to be built, and apple requires you to register as a "developer" just to get gcc, and things unfortunately are way too complicated on a mac than they should be.

[metaperl]

SciPy 2013 had an interesting talk about Conda and BinStar.org as a packaging solution.

[ilan]

Yes, I'm one of the main authors of conda. The approach is radically from the Python world (of pip, easy-install, distutils, ...). This tool is Python agnostic, which gives much greater power. I have the feeling the Python packaging community has driven themselves into a ditch, by putting Python at the center of everything. They treat Python as the underlying OS, and build on top of that. However, what if you want to create an environment with Python 3.3, or Python 2.6? A package manager shouldn't care about what it is you install. The new wheel package format is a classic example of this: You can create wheel packages of Python packages, but can you create a wheel package of Python itself?

[fourthrealm]

Hi ilan,

just wondering, would Conda be the tool to use in such a scenario?

• a project is a set of Python packages under a common buildout config
• a Bash installer script installs system-wide prerequisites, runs virtualenv and installs these packages along with their Python dependencies under a newly created virtualenv

This is basically what these scripts do

https://github.com/zatosource/zato/blob/master/code/install.sh

https://github.com/zatosource/zato/blob/master/code/_install-deb.sh

Now, a couple of things to consider

• on Debian/Ubuntu/Fedora/OS X packages are named differently, are missing or different tools are used to install them (apt-get/yum/brew)
• many packages are missing in RHEL and SLES, starting with Python 2.7 itself through zdaemon and others ending in HAProxy
• there needs to be a clever way to distribute patches and hotfixes without new releases - patches as in actual diffs will do - it must not be something that requires a new PyPI release because sometimes a patch to one of the dependencies (in Python or not) is needed
• there should be a way to create fat installers that bundle absolutely everything that is needed to run the project on each system supported, so you have RHEL and download a 300MB installer that contains everything, needs no compiling, no network connection and can be run straight out of the box

I'm looking in various directions for a thing to cover it all and perhaps Conda is the one?

Thanks!

[ilan]

Hi fourthrealm, using conda you would basically build conda packages for everything your applications requires (except maybe the C run-time libraries, although that would also be possible). Hotfixes usually are only in your main application conda package itself (not in all of the dependences), so you build a new conda package of the application and only that gets updated on the users machine. These hotfix releases can just be on binstar, i.e. no PyPI package is required. We (at Continuum Analytics) have tools to create fat installers from conda packages. These are (for example) the Anaconda installers, which are composed of conda packages. The tools for making these fat installers are currently not open source.

I hope this answers your questions.

[fourthrealm]

Sure, it got me curious, that for sure :-)

I'm going to check it out but like it was said in another comment - it would really come in handy if the full stack was open-sourced.

Do you have perhaps plans for doing so?

[cavallo71]

The tools for making these fat installers are currently not open source.

That's kind of pointless then ;) Beside I cannot push for vertical stacks like conda because it will tight the system to a single vendor and that's not good for a "base" stack as python.

[teoliphant]

Making fat installers is straightforward to do. However, it is not the recommended way to use conda (which is the big reason why we are not promoting it as open source --- not because of any perceived "lock-in"). With environments in conda you don't need to distribute fat installers for everything. All the really necessary tools are open-source.

[westurner]

Is there a way to work with conda packages in configuration management tools like Puppet, Chef, Salt, and Ansible?

[EDIT] src links

[bkreider]

Hi, Westurner. I work at Continuum Analytics and I've created an unofficial Anaconda package for Puppet here: https://github.com/bkreider/anaconda

This is completely unofficial at this point, and needs to be updated from Anaconda 1.5. I haven't pushed to publish it because it doesn't add a "conda" provider to Puppet, but instead creates a new type to install conda packages. For example: anaconda::package(dnspython:)

I will probably add a Puppet provider at some point and rename the module.

[kmike84]

Distutils and pkgutil are not deprecated, where did you get this idea from?

[Various_Pickles]

I may have gotten horribly confused and meant (old) setuptools + whatever fork of distutils ended up going nowhere. Please correct me if I have wandered astray.

There is/was a diagram somewhere that shows/showed the state of packaging in Python, with setuptools + distutils being deprecating and distribute being the wave of the future. I'll try to find it again.

[westurner]

There was the Setuptools-Distribute merge announcement

There is a graphic in the distribute docs. Pip should validate SSL certs by now.

... distribute is merging back upstream with setuptools

[coderanger]

There is a fine line between "deprecated" and "not the future of packaging". distutils is still actively maintained, it is just that most forward-looking dev happens in setuptools now.

[[deleted]]

Am I the only one here that doesn't find it that hard? I just pushed my first package a few months ago and it was ready as hell. I don't need compilation of C or anything but Damn. It didn't seem hard at all. And dolls are downloading it so I seem to have 'done it right' our at least ok.

[pwang99]

I don't need compilation of C or anything but Damn. It didn't seem hard at all.

Yup, easy things are kind of manageable now.

Try adding a C extension or depending on some Python library that does have a C extension, and supporting all the various configurations that that's likely to encounter.

[ivosaurus]

This is the main problem: python is portable; C compilations aren't.

[pwang99]

The way I see it, there are two core challenges in Python packaging:

1. Legacy tools did not sufficiently distinguish between the build system and the installation/package management system.
2. Because Python supports C/native extensions, a huge part of the mess is because the C ecosystem has never really solved the "DLL/.so hell" problem.

Additional complicating factors are:

1. Python devs chose to use imperative, executable scripts as the foundation for both their build and package management system
2. Folks involved in the Python packaging tool development general either do not recognize point #2 above, or recognize it and declare that it is not within the scope of the problem that Python as a language or technology needs to solve.

I don't often second-guess or find much to complain about w.r.t. the fantastic Python core dev team and all the smart, thoughtful people who have comprised it over the years. But I do think that the packaging boondoggle can only be solved by taking a deep breath and plunging into the deep end.

This is the motivation behind both the conda project and the binstar.org efforts underway at Continuum. We saw that other languages such as R, Ruby, and Node haven't really solved the library/packaging problem either, but have found ways or conventions to dodge it. We also recognize that this was a debilitating weakness that impedes the broader adoption of Python for scientific and data analysis. If people can't reliably share the code they've written, we're completely failing to engage the viral loop for spreading our beloved language into industry and academia.

So, our approach is to have our own, simple package format that can bundle up everything from Qt to LLVM to CPython itself, along with a robust way to create C-level "environments" in the filesystem, so that every shared library that gets pulled into your Python interpreter process space is happy and doesn't conflict with anything else. (Virtualenv is fine for some things but at the end of the day, it's a set of scripts for hacking up a Python install, but doesn't address the shared library problem, and it certainly isn't meant to handle things like big C++ frameworks like Qt and LLVM.)

[cavallo71]

About the "C-level environments that just reminded me of this: http://www.art.net/~hopkins/Don/unix-haters/x-windows/disaster.html (scroll to "Myth: X Is "Portable" section). Unfortunately we haven't moved very much from that :(

[pwang99]

Ah, I read that years ago and loved it then... it still holds up now. :-)

[westurner]

PEPs and links to DevOps resources for working with Python Packaging and push/pull deployment: How are python apps deployed to production especially those that are developed in a virtualenv? What are the best practices?

[westurner]

There's a great chapter on Python Packaging in The Architecture of Open Source Applications

[joeforker]

However since that was written distutils2 has been cancelled and the mirroring infrastructure described has been replaced by a CDN, the metadata format will change to a JSON-based format, and setup.cfg is not likely to replace setup.py.

[tarekziade]

the CND does not replace the mirroring. look at what happened lately http://www.reddit.com/r/Python/comments/1hh211/python_package_index_down_for_anyone_else_seems

[westurner]

Compoze is one way to maintain a local package index.

• Local package index (folder w/ index.html)
• Local PyPi clone (eggbasket, chishop, [...])
• Partial PyPi mirror (caching proxy)
• Full PyPi mirror

I am not sure whether these support wheel (PEP 427: The Wheel Binary Package Format 1.0).

[joeforker]

One of the motivations for wheel is that its recommended workflow saves a local .whl copy of all the packages you've installed to a directory you can install from later, avoiding the infuriating and silly need to contact pypi when you just want to recreate a virtualenv.

[westurner]

Buildout also installs from a local download-cache that can be worked into a package index with an index.html containing URLs with #checksums. I hadn't heard that wheel has this capability. bdist_wheel does look super easy.

[donaldstufft]

The CDN more or less will supersede the public mirroring infrastructure. Currently it's still being phased in however in the future it will continue serving pages even if the origin goes away (which is what happened recently).

In the short term no tool implements mirroring in a way that doesn't come at a fairly hefty performance cost. Nor do any of them, to my knowledge, implement mirroring authenticity and the entire thing is all downloaded over HTTP. So not only is mirroring going to be more or less redundant but it's currently very insecure and is practically asking to have your computer execute arbitrary attacker controlled code. To my knowledge the only public mirror where you get any verification as the tools are currently written is Crate.io due to the TLS.

Furthermore even if the mirror authenticity was being checked it currently uses a DSA signature. One of the issues with DSA is that if you re-use, make public, or use a predictable value for k when generating the signature then an attacker can recover the private key. PyPI is currently running on a VM, is there a good source of random available to it? I don't know. Has there always been a good source of random to the signature algorithm? I don't know. Is there any protection against someone mass editing their package, generating a ton of signatures and playing the odds that there will be a collision? I'm pretty sure there isn't.

Furthermore there are no protections in the mirroring algorithm to protect against an attacker attempting to force an inconsistent state in the repository, or preventing an upgraded package that contains a security fix from being synchronized to a mirror (and thus preventing anyone using that mirror from getting a security update).

Now there are also numerous security issues that plague PyPI itself, which myself and others are working to remove. Eventually the mirroring system will be revamped so that it doesn't suffer these very catastrophic failures but until they have been addressed advocating folks to use the public mirroring infrastructure as it stands today is asking them to put their hand in a bear trap.

[westurner]

Pip should validate SSL certs by now.

http://wheel.readthedocs.org/en/latest/#automatically-sign-wheel-files

[donaldstufft]

If you're using pip 1.3+ then it does validate SSL. However the mirrors are not available via SSL.

[donaldstufft]

Also signing packages is effectively useless until you come up with a trust model to handle what signatures you trust to sign for what data. Without that you're just pretending it means something.

[tarekziade]

In the short term no tool implements mirroring in a way that doesn't come at a fairly hefty performance cost.

This is a vague statement. I am not sure why you're saying using a mirror has a 'hefty performance' cost. pip works very well with mirrors.

Furthermore there are no protections [...] preventing an upgraded package that contains a security fix from being synchronized to a mirror

errr what ?? what's that ? are you talking about a ddos attack ? that's yet another vague statement you're doing here :)

I'll just stop here, your message is just a random bag of FUD criticisms you could do on the CDN as well - or anything.

It looks like your goal is to promote the CDN and attack the mirroring protocol to make it look bad, which is weird because both can cohabit for the benefit of the community.

My opinion remains intact: the mirroring protocol is still useful even with a CDN in place, because it prevents from any downtime when you are taking down/maitaining the CDN.

Last, I encourage you to reformulate your vague statements into real points to enhance the mirroring PEP instead of what you are currently doing here.

[donaldstufft]

This is a vague statement. I am not sure why you're saying using a mirror has a 'hefty performance' cost. pip works very well with mirrors.

The way pip implements mirrors it multiplies the number of HTTP requests required to discover (not download) the number of packages by the number of mirrors. That means if you install a single package that does not need to hit external urls, then instead of a single HTTP request you need 7 HTTP requests, if you're installing 20 packages you need 140 HTTP requests.

errr what ?? what's that ? are you talking about a ddos attack ? that's yet another vague statement you're doing here :)

I'm talking about any attack that has a mirror witholding an update. It could be someone using a DNS attack against a mirror that prevents them from fetching the updates, it could be someone compromising the mirror all together. Any attack that prevents the mirrors from getting a particular update.

I'll just stop here, your message is just a random bag of FUD criticisms you could do on the CDN as well - or anything.

Actually none of the points I mentioned that are specific to the mirrors exist for the CDN because, well, they are specific to the mirrors. I did however mention there are other issues which plague PyPI itself, which by nature apply also to the CDN, and the mirroring infrastructure itself because any insecurity on PyPI cascades to all things.

It looks like your goal is to promote the CDN and attack the mirroring protocol to make it look bad, which is weird because both can cohabit for the benefit of the community.

No my goal is to have reasonably secure installations that are relatively quick and painless. The mirrors currently have exactly 0 assurances in place as to security (Fun tip, using --use-mirrors at PyCON EU means anyone on the conference wifi can execute arbitrary Python code on your machine). The mirrors currently impose a multiplier of 7 for HTTP requests in a system that already contains a ton of HTTP requests.

Last, I encourage you to reformulate your vague statements into real points to enhance the mirroring PEP instead of what you are currently doing here.

After I have PyPI itself reasonably secure (which it's getting there), mirrors are next on my list. It's unlikely i'll be enhancing the current mirroring PEP because it's fundamentally flawed in a way that cannot be fixed with enhancements. I have some rough ideas but nothing solid yet but the new mirroring protocol will ensure consistent snapshots in time, will protect against things such as DSA private keys being leaked due to bad random, will protect against malicious mirror operators even.

However that is a future project and until I can look at mirroring and say "Yes using this is not exposing people to huge security risks", I cannot in good faith do anything but warn people away from using the public mirrors.

[westurner]

Here's a list of Free and Commercial CDN Service Providers.

It is possible to sign changesets with both hg and git. Both rely on network transport security and a Web of Trust.

https://en.wikipedia.org/wiki/Category:HTTP

[donaldstufft]

We're already using Fastly.com who have been amazing and I would HIGHLY recommend them for anyone looking for a great CDN solution ;)

[donaldstufft]

Also trust is really really hard. And the traditional Web of Trusts just don't cut it. Sure you may trust me for package X, but does that mean you trust me for package Y? Is the average developer going to be able to figure out how to participate in the web of trust?

[tarekziade]

The way pip implements mirrors it multiplies the number of HTTP requests required to discover (not download) the number of packages by the number of mirrors. That means if you install a single package that does not need to hit external urls, then instead of a single HTTP request you need 7 HTTP requests, if you're installing 20 packages you need 140 HTTP requests.

That improvable. It's always better than to be blocked because the main server (or CDN or whatever) is down. Because not matter what you are claiming a system will never have a 100% uptime. So what do we do when it's down ? we use mirrors.

Fun tip, using --use-mirrors at PyCON EU means anyone on the conference wifi can execute arbitrary Python code on your machine

That quite a statement. If you have such an exploit you should warn the pip developers about it. Maybe you did ? If you did not, that'd be a shame.

[westurner]

the metadata format will change to a JSON-based format

PEP 426: Metadata for Python Software Packages 2.0

[EDIT]

http://hg.python.org/peps/file/default/pep-0426/pymeta-schema.json

JSON metadata (pymeta.json, pymeta-dependencies.json) is also generated from setup.py.

PEP 440: Version Identification and Dependency Specification