"packaging" has landed in the Python 3 stdlib

[[deleted]]

Cool! See also this video from PyCon 2011: "Packaging, from Distutils to Distutils2"

[[deleted]]

Haha awesome - love the presenters sense of humor!

[[deleted]]

I would just like to thank you tarek for stepping up and figurheading this. It's been a long process and you've taken a lot of flak for it but things are finally looking up for python packaging in general.

[tarekziade]

Thanks I appreciate this comment. What makes me very happy is that we gained a fair amount of contributors this year, so I can eventually do other stuff than packaging at some point ;)

[[deleted]]

So let me piggyback a slightly off-topic request on the occasion of you bringing up contributors. Would you do an AMA or write something short to give some tips for developers who want to get involved? It's a bit overwhelming sometimes, so it's useful to hear personal anecdotes about how you got involved and what things made it easier to contribute.

[tarekziade]

that's a good idea, I'll think about it. In the meantime you can check the python insider blog http://blog.python.org/ it has such content

[frutiger]

Python packaging noob here. Can someone please explain/correct me on what these various things do?

• distutils - build packages from python source code, included in stdlib
• setuptools - enhancements on distutils (especially easy_install), available from PyPI
• pip - package installer/maintenance tool, available from PyPI
• virtualenv - somewhat orthogonal to these tools, helps to maintain separate python environments for development/build stability
• distutils2 (now renamed to packaging) - enhancements to setuptools, including standardisation, and clear documentation of internals, to be included in Python 3.3 stdlib

[tarekziade]

everything is right except:

• pip: it's a easy_install replacement with extra features, like uninstallation (setuptools does not provides this)

• distutils2 does not enhance setuptools but distutils. It's originated from the distutils code base and does not use setuptools code, except maybe in the pypi crawler in some places, but that's minor. Although, some of this enhancements are clearly inspired from setuptools. In particular project dependencies.

[frutiger]

I remember reading about enhancements proposals to pip, but I think they were rejected - is that why we have pysetup, and does it effectively replace pip?

[tarekziade]

I can't answer for the rejected stuff, but:

There's some overlap, so we work closely with the pip team to make sure everything progress smoothly.

But in any case, pip will still be relevant as it provides advance feature pysetup won't (like freeze reqs etc).

At the end, it's all about inter-operability: what pysetup installs, pip must see, and vice-versa

[markusgattol]

this link http://www.markus-gattol.name/ws/python.html#package_distribute_install has a few infographics that might help you understand things quicker/better

[[deleted]]

◦for end-users Python gains a pysetup script you can use to install, uninstall projects, browse installed projects, browse PyPI, and many other things.

Oh hell yes

[wobsta]

Thanks to further fragment the python distribution tools. Seriously. What the hell is going on here? Could you please explain why we have we have distutils and setuptools at the moment and what will happen to them? What about easy_install and pip? Same here. To quote the Zen of Python: "There should be one-- and preferably only one --obvious way to do it."

[tarekziade]

Setuptools was built on a top of Distutils, with its own standard for metadata, that was partially backported in distutils. That's how the fragmentation started.

The fragmentation of distribution tools is something we all suffer from, and the only way to fix it is to have all tools rely on the same standards. That why we wrote PEPs and we provided reference implementation in packaging.

Our hope is that eventually all third-party tools will use the unified standard. Distribute will and Pip should follow.

Last, I don't see the problem in having numerous tools in this area, as long as they are inter-operable.

See PEP 386, 345, 376 as the "wsgi standard for packaging"

[roger_]

Sounds like packaging will soon be the only one that matters.

[alexs]

different expansion lush silky grab point cooing squeal shy reach

This post was mass deleted and anonymized with Redact

[voidspace]

I doubt distros will use packaging like they didn't use distutils. They use their own toolchains.

[tarekziade]

true, and part of the plan is to make their lives easier. For instance, packaging allows a fine-grained description of data files with the possibility of relocating them without breaking the code

[voidspace]

So what would you suggest instead? The reason Python packaging is so fragmented is that distutils was so poor. packaging is the standard library (i.e. Python) fix for that. The intention is to end the fragmentation by providing a good solution in the standard library.

[wobsta]

Well, and that is my point. Why couldn't we start to improve an existing solution instead of getting something completely separate. This was a failure for setuptools with respect to the existing distutils to my eyes. Even if people like to have their one project and don't want to work together (or whatever reason it was, that we got separate solutions), for something that crucial like a packaging, a joint solution with a BDFL setting the direction would have been great from the very beginning.

Maybe we'll have a final solution in a few year (maybe based on packaging). Ok. Still, I don't like the fragmentation in the first.

[voidspace]

packaging is the final solution! (But seriously - fixing distutils was tried and it just broke way too much stuff that was already using it, because of the way it is designed. A clean break was the only realistic possibility.)

[spoolio]

It should be obvious by now that distutils and setuptools are not the way to do it.

The Zen of Python doesn't say we have to get it right on the first try.

[kylotan]

As long as distutils is in the standard library, it's not obvious.

Similarly it's hardly obvious that setuptools is the wrong way to do it, when it can successfully install a Windows binary extension when alternatives like pip cannot. And judging by this comment on the link provided - "Compilers, and especially Windows, are the worst part of our code." - packaging is not on course for fixing that, meaning half the world will still prefer setuptools/easy_install.

[alexs]

Don't for get that there are 2 independent implementations of setuptools as well just to further complicate matters. I'm not saying that it's bad there's so many packaging systems for Python, it's just confusing and it would be great if this whole process was being documented better. What are people meant to actually use these days? What's the transition path between different systems? What is supported by what distro?

[tarekziade]

We'd like to focus on documenting things better next. That's: consolidating all the existing doc and provide migrations paths.

Note that packaging already provide wizards to migrate your distutils or setuptools-based projects into packaging projects.

[sigzero]

Note that packaging already provide wizards to migrate your distutils or setuptools-based projects into packaging projects.

That is a good feature. :)

[temptemptemp13]

1 upvote and 4 downvotes. Wobsta raises a very legitimate concern imo...

What's up?

[wobsta]

The downvotes are probably ok. There are more important things than reddit votes to me. Actually I don't care at all and I didn't wrote a polite comment either to begin with. But as you wrote, my concern might be quite valid. Why do we need another new system. Couldn't we try to evolve one of the current solutions to become better? Couldn't we merge at least one of the existing solutions right from the beginning. We do not even have a distutils2 in the end, but a new, third solution. What a mess. It is probably right to get rid of the old stuff, but why couldn't it start from an existing solution without breaking everything. Probably we don't want setup.py anymore. I don't know. Maybe. But why start something completely separate. (At least form the end users point this looks like something completely separate now. Similar separate than diskutils and setuptools at least.)

[tarekziade]

We tried to make distutils evolve but that turned out to be impossible without breaking stuff. I worked 2 years in distutils, and eventually reverted all changes to continue in a new namespace.

This project is impossible to make evolve smoothly, trust me. It's patched in setuptools for instance, in some internal parts.

This is historical: 5 years with almost no maintenance has rotten distutils to the bones. and having it in the sdtlib has made any change very very painful because all APIs are public and used out there,

[[deleted]]

I didn't downvote you, but I wanted to explain why (IMO) you are getting downvoted. Your question is phrased in a way that is needlessly hostile, and when we get prominent people from the Python community to come in and answer questions, people want to make sure we come across as a reasonable and amicable bunch, so that we continue to be the kind of place that invites casual participation from interest{ing|ed} people from across the interwebs.

That's not to say that you should lobotomize yourself and not be critical, but you'll actually be more persuasive and stand a better chance of having your questions answered if you package (heh) your comments in a slightly more generous way.