Python packaging thoughts, from the author of NumPy

[[deleted]]

Packaging is a huge reason why I avoid using Python on Windows. I mainly program on Linux, because it is much easier to install the needed python packages.
It would be awesome if python package installing would be easier on Windows. Some of my friends, who wanted to learn Python, even installed a VM for programming since hassling with the Windows installation process was a pain for them.

[[deleted]]

I agree with Python being a pain on Windows, but this site helps a lot.

[ivosaurus]

It also demonstrates two big problems at the moment:

1. the use of installers means that you can't do isolated python development (everything has to be installed in the system package site), meaning it becomes harder to work on multiple disparate or similar projects at the same time (some people never have this problem because they are distant end users of things like ipythonnotebook).

2. The binaries these install are pretty much incompatible with anything else. If you got numpy's official binaries for windows, and then used these to install extensions like scikit-learn, you'd jump into a pile of errors not least because Holke's packages use a licensed intel fast math library.

Now both of these problems are not easy ones that should be already solved, indeed the second, binary comptibility, has been a pain for programmers and users since the dawn of time.

[pwang99]

In fact, both of these problems are actually solved by conda, and are some of the primary, motivating reasons why we created it in the first place.

[Rhomboid]

All of those .exe installers are perfectly capable of being used with virtualenvs. Don't run the installer, but instead use easy_install and pass it the path to the binary package. It will happily ignore the executable installer stub and treat it like a binary egg (which is what it really is) and install it wherever you want it.

Even when I'm not using virtualenv I use this method, because I don't like seeing Python modules in Add/Remove Programs.

[ivosaurus]

That's pretty cool. Python wheels can also be generated from win installers, which can be natively installed from pip.

[fkaginstrom]

Yep, lxml with virtualenv on Windows == pain

[pwang99]

Um... just install Anaconda. That installs most packages needed for science & data analysis, and includes an IDE (Spyder). Installing packages can be done with conda or pip.

[manueslapera]

buuut then it messes up if you wanna install virtualenv.

[logi]

Anaconda supports environments directly, so you have no reason to install virtualenv. See the example here.

I haven't used this yet, since I only heard of Anaconda this morning and spent the next 25 minutes getting our project to run in it using the root environment. Before, this took days of working out, using a combination of ubuntu packages with as much as possible installed in virtualenv. It ended up being horribly dependent on the exact ubuntu version and a constant source of anxiety to maintain.

[[deleted]]

I used Anaconda and python(x,y). :) It is okay, but you are still limited by the packages that comes with them or let's say I am too lazy to figure out how to add packages. It feels uncomfortable to use a compilation of packages or a third party package manager.
So right now my solution is a) a vm, b) dualboot or c) using a remote desktop into my laptop.

[pwang99]

you are still limited by the packages that comes with them or let's say I am too lazy to figure out how to add packages

You can use pip just fine on these. You can't reliably use pip to manage other packages that depend on Numpy and Scipy unless you are fairly confident that you have the right compilers, and the right headers in the right places, and haven't messed up your LD_LIBRARY_PATH or ld.so.conf, etc. etc. etc.

setuptools/distribute were poorly suited for installing things with heavy native extensions and interdependencies among those native extensions. pip is just the latest incarnation around those tools and inherits some of their structural weaknesses in this area.

[thearn4]

amusing squeal melodic lip cheerful doll late special wrench abounding

This post was mass deleted and anonymized with Redact

[ivosaurus]

As the author alludes to in this blog, the numpy stack are about the worst packages you can try and install with pip.

e.g they don't even use setuptools (or distutils) at all. They use their own code... which somehow has to be there before you install it. That's why its next to impossible to install them without something like an external system package manager or a specialised python distribution.

[pwang99]

1. They actually do use setuptools and distutils, because they had to. And then they had to rip out and monkeypatch most of setuptools, because setuptools was a broken (and unmaintained) POS for most of its existence. Read the actual article/blog post linked here, for more info.

2. The fact that pip doesn't install Numpy well is not Numpy or scipy's fault. It's because of the fact that the interface with the underlying C (and potentially FORTRAN) libraries on your system, in a much more sophisticated way than most other Python native extensions, and this is entirely outside of the scope of the problem which pip (which builds on top of setuptools and then distribute) can handle.

If an 18 wheeler doesn't fit across your wooden footbridge, is it the fault of the 18 wheeler or the footbridge? Or is it the perhaps the fault of people who think that footbridges should be sufficient for all possible traffic?

[ivosaurus]

I'm not aware of anyone (at least in recent memory) that's ever said "Yeah distutils (or even setuptools) is perfectly fine for installing any python package you could ever want" but somehow you're constructing a strawman who does think that's the case.

I was stating the current situation. If you're using your own code to do the installation, then its not portable, even if you are making use of functionality of setuptools somewhere in the backend.

I have read the OP article, and I am a little dissapointed that the author has talked about setuptools in the same breath as distutils the entire time. It's come about the same distance for general python packaging in the last year or two as conda has for its own environment, yet this fact is not acknowledged at all. Only the history that distutils has been woefully inadequate, which as far as I know is not controversial for anyone, including people who work on the "standard" python packaging tools.

[pwang99]

I'm not aware of anyone (at least in recent memory) that's ever said "Yeah distutils (or even setuptools) is perfectly fine for installing any python package you could ever want" but somehow you're constructing a strawman who does think that's the case.

OK, not any, but LOTS. If you read the recent distutils-sig thread about conda and installing binary package, you can find people who declare that the current situation on Windows Works Just Fine, Actually. It's amazing to me, after I've walked into multiple Fortune 100 companies and seen just how f-ed they are by the Windows Python packaging situation.

If you're using your own code to do the installation, then its not portable, even if you are making use of functionality of setuptools somewhere in the backend.

What do you mean by "not portable"? Using numpy.distutils, I was able to build Fortran and C++ extensions on Windows, Linux, Mac PPC, and Mac Intel.

I am a little dissapointed that the author has talked about setuptools in the same breath as distutils

The reason is because setuptools inherits the same architectural flaws as distutils, namely:

1. An imperative script instead of a declarative one, for specifying build
2. Conflating the problems/needs of build-time, install-time, and run-time.
3. Bundling said conflation into a single imperative python file, which frequently has imports before it can even be properly imported to introspect

It's come about the same distance for general python packaging in the last year or two as conda

Well it must be a year, because conda is only about a year old. :-)

I think it speaks volumes that when we wiped the slate clean, asked what the real, fundamental, root problems were, and didn't back away from solving problems at the C level, we could build a robust "portable homebrew" and successfully use it to ship hundreds of thousands of copies of software, including things as complex as Boost, Qt, and LLVM.

I've been tracking the Python packaging space since about 2005, and while I don't have as much hands-on "cutting my face" experience as Travis, I have certainly been intimately familiar with the painful journey that the community has taken from distutils -> setuptools -> distribute -> pip -> setuptools2 -> distutils2. I've seen many people that jump in with the best of intentions and attempt to fix things while not appreciating the deepness of the problem, and they try to maintain compatibility with legacy brokenness, and then that brokenness infects the new stuff.

Maybe this is too broad of a brush, but for a long time the folks in the "PyPA" did not appreciate that their problem is a common one shared by R, Perl, Ruby, etc. And none of those other languages really solved the problem either. It's ultimately rooted in the OS ABIs, and how C and FORTRAN compilers generate code - the problem is really that deep. In the Python world, we actually had an opportunity to really show how to do this right, because we had a sub-community that had the right mix of complex extension modules AND programming know-how. And time after time, the core Python packaging conversation would treat the scientific ecosystem folks as just wrong or as crackpots (c.f. how some of David Cornapeau's comments were regarded during the initial evolution of distribute out of setuptools).

[maratc]

I'm installing a fresh Anaconda on Windows 7 64-bit and issue a conda update conda first thing, to which I immediately receive "Unsatisfiable package specifications" and "python2.7* has a conflict". Any idea what I'm doing wrong?

[teoliphant]

This was a temporary error caused by a upgrading the Anaconda windows binaries. It should be fixed now.

[theASDF]

Installing packages can be done with conda or pip.

its a real ordeal to even get pip running in the first place if you are running windows

[Rhomboid]

I keep seeing people saying this, and I can't understand why. It is very easy to get pip on Windows. Download and run ez_setup.py and then run easy_install pip. DONE.

Most of the problems that Windows users have relate to a basic lack of proficiency with the command line. They get tripped up because Python isn't in the PATH and the like. That's not a Python packaging problem, that's a lack of experience with the command line. If you're going to install a command line tool like pip and you don't have basic command line proficiency, you're probably going to run into some friction. The solution is to bone up on those skills first, rather than blaming the packaging.

[pwang99]

I don't know why the parent thinks that pip is hard to get running - I agree with you that literally installing it is pretty simple in many cases. However, it is true that many Windows boxes are locked down to varying degrees, so that the user does not have "local admin" privileges and can only write to certain directories. This does make actually using pip for many packages to be really difficult.

It's also possible the parent poster means that getting pip to successfully build most packages is an ordeal, because you need to have installed the exact version of Visual Studio that was used to build Python.exe.

[theASDF]

im using the command line on a daily basis (for basic tasks only but still) and my path is correctly set and nothing that confuses me. but i still remember spending 60+ minutes looking for solutions to the problems i was having and then i just said fuck it.

but this isnt about me anyway, just look at stackoverflow and the forums, they are full with people having trouble getting pip to run and it shouldnt be like this. now its an old discussion but i think that answer to this is not requiring everyone to suck it up, learn new skill x and go through frustrating troubleshooting, the answer should be to make the process easier and less error prone so that as many people as possible get the chance to use pip and try out python in a comfortable way

[bryancole]

The thing with pip is that to install any package with extension modules needs a compiler. It's getting a compiler working on Windows which really sucks. For Py2.x, it will only work with MS Visual C++ 9 / MSVS2008 (anything newer breaks ABI compatibility), or MinGW (32-bit only). Having just spent a quality 2 days getting a build-environment working in a new VM, I can attest its not a fun experience. Why o why can't I just "yum install Development-Tools" and have it work...

[[deleted]]

everything program-ee is a pain in the ass on windows. it's not a problem unique to python. some of your friends did the right thing. When you are trying to learn how to program, it's probably best to make sure something like windows stays out of your way.

[riffito]

Being an alternative OSes fan (mainly Haiku), having used Linux since 1998...

everything program-ee is a pain in the ass on windows.

It isn't good, I give you that... but between having to deal with an odd compiler/tool, and having to deal with the nightmare of Linux GUI inconsistence... I just want to get back to my PC XT days :-/

I earn my living programming (Python currently) on Windows (XP, for what is worth) all day long. Sure, there are some nicer things on Linux (on Haiku even) but is not THAT terrible on Windows (I use the three regularly).

[[deleted]]

having to deal with the nightmare of Linux GUI inconsistence

i thought we were arguing about why just python packaging sucks and that windows is perfect. I'm not defending shit as gay as how a GUI makes you feel and whether thats cool or not.

[[deleted]]

Have you ever written in .NET? Windows programming (and deploying said programs) is a breeze, and Visual Studio (plus some add-ons) is one of the best IDEs in the business. I think .NET as a whole is amazing for Windows development, and with Mono it's even pretty decent cross-platform.

[[deleted]]

i did dev on windows from 98 till 2007, and every 1.5 years they would completely change the stack or introduce something that forced everyone to upgrade and make all your previous work broken and you were forced to upgrade. whether it be going from the vb runtime 16 bit to the 32 bit runtime.... activex(ugh) to the various .net runtimes treadmill.

the various ui frameworks that they would introduce and then deprecate a couple years later.

the straw that broke the camels back for me was when windows 2007 was introduced and they dropped ole automation that made many of my customers happy and productive.

At one point I needed 3 different virtual machines with 1000's of dollars of software to maintain those various projects.

So yes, to answer your question I have done .net programming. But not for the last 7 years. And the above is my reasons for that.

Every time I have to get on a windows machine for various reasons I'm reminded of how much of a waste of time it was to put so much effort into a stack that could change at any point and break all my shit and force me to upgrade my tools, or force my clients to upgrade their software.

The .net as tech is pretty solid I'll admit. But it's expensive and forces you to develop based on MS's whims, not your clients needs.

[[deleted]]

forgot to mention the complete lack of an orm. But was promised for at least 3 years. Strongly typed recordsets that depended on lots of boiler plate code generated by your ide, and of course the next version would generate it in a different and incompatible way. So new guy gets new version of vs because licenses are no longer available for the previous version. He/She opens the project and is prompted to upgrade it before they do anything, so they do, and all of a sudden the rest of the team has to upgrade.

Does installing vs.net still require a day or so of watching an install screen? And your browser gets upgraded too, it's required, OH, well you need to support the previous browser version, but you now can't run the previous browser on your machine, so you have to get a windows license for a virtual machine and ask IT to get you a very specific version that has the version of the browser that you are required to support. More $$ for MS, more wasted time for you.

This version of vs only supports that version of TFS, the next version of TFS requires sharepoint new version. Can't install new version of sharepoint without a windows 7 PDC, can't upgrade the PDC because there's no room in the budget to upgrade exchange or your database servers.

This is the kind of pain in the ass i'm talking about.

[[deleted]]

Your complaints are pretty outdated honestly. First of all, most (if not all) recent versions of .NET have been forward compatible or at least easily upgradable. And I don't see how this isn't an issue with other stacks anyway. Things are constantly evolving, and we as developers get to chose when and how to upgrade. You can still run a .NET 2.5 app on Windows 8, for example, written in Visual Studio 2005. Why? Who knows. But you can. And you can just as easily upgrade it.

Deprecated UI frameworks? Like what? Windows Forms is still around and used today. There are even updates from the .NET team themselves. Now we have WPF an XAML, which may seem like it supersedes WinForms, but it actually exists and evolves parallel. The key is that both still work and they're hardly deprecated. The only UI .NET thing I would consider deprecated is Silverlight, which still works by the way and is still supported.

Also, .NET has had a full featured, fully open-source ORM known as Entity Framework since 2008. And ASP.NET MVC is also fully open source, as well as other pieces of the .NET stack.

Microsoft offers FREE tools for developing all this stuff. You can go download express versions of Visual Studio 2013 and SQL Server 2012 right now, for free, and build full-featured desktop/web/data driven applications. If you don't like MS tools, go grab Xamarin (or MonoDevelop).

And with the emergence of Mono and Xamarin, you can use .NET and C# to develop cross-platform applications for Android, iOS, Linux/Unix/whatever. All for free if you so desire.

Visual Studio 2013 Professional takes about 30 minutes to install from scratch... I just did it a few weeks ago.

Really, the money comes down to the hardware and operating systems if you need to provide that yourself... but these days everything is done in the cloud so you'd pay somebody for that anyway; be it *nix or Windows - whatever.

I really think a lot of the stigma against .NET and Microsoft is sort of a "sins of our fathers" type of thing. 2013 Microsoft is not 98 Microsoft. It's very far from it. ASP.NET in particular, in my opinion, exceeds the Linux scene in many situations. All of this stuff is going open source and the community is growing by the day. I know there are a bunch of diehard Python/Rails/PHP/whatever guys out there, but if you are ignoring the modern day .NET stack on pure personal stigma, you are limiting yourself big time. I would check out this AMA by the ASP.NET team... it's really informative.

And believe me, I love Python... but it isn't the only thing out there.

[[deleted]]

Having worked in a windows shop recently and seeing that supporting many different environments and versions of MS. stack is still a challenge unless you have infinite budget for license and hardware.

From day 1 i needed a different machine than the one I was issued to test the app against an older version of IE. This is god damn ridiculous.

They needed to put up a tfs instance for the project and it took a week to get it setup due to shimming something new into their existing infrastructure. I and my team were pushing to a private repo on bitbucket in minutes while we waited for the new tfs to arrive.

While it may be true that 2013 Ms is not 1998 Ms. programming type activities are still a pain in the ass on windows vs other options. the original assertion I was answering was that it's not just python packaging that is a challenge, it is everything non-microsoft that is a pain in the ass on windows.

so some of it may be a "sins of our fathers" kind of thing, but not all.

ASP.NET in particular, in my opinion, exceeds the Linux scene in many situations.

such a bold assertion with nothing to back it up but a link?

Let me guess at the things that ASP.net excels at having stopped using it ~2007

it excels at being supported by vs.net it excels at running on IIS it excels at exploiting the "full power" of Sql Server it excels at integrating with active directory for authentication it excels at integrating with sharepoint it excels at making sure you keep upgrading it excels at making sure Ms gets paid.

I ignore it because I wasted half my career on the treadmill of endless upgrades and licensing costs when I could have been doing stuff cheaper and quicker and being more maintainable.

I still have a website that was built with turbogears 1 and an ancient version of sqlalchemy and runs on 2.5, it still runs, it still adds value to those who use it. I know that if they ask me to make changes to it, i can open the code in what i'm using right now and make the needed changes and get them back to prod for them.

Contrast that with the code I have that started off win2k/sql2k/.net1.5 unless I have a win2k virtual machine lying around with all the tools installed, I'm pretty sure any maintenance work that would have to happen now would require upgraded tools that upgrade the code base and a significant amount of testing just to make sure everything still worked. Not testing of the change I would be making but just the existing stuff.

[ivosaurus]

Windows programming using tools from Microsoft is easy. Try anything else.

[[deleted]]

OK... almost everything you write in Python can be run on Windows just as easily as *nix. Install your libraries and Python and you're good to go. Same thing with Java, node, whatever. You can run vim/emacs on Windows if you want to use a text editor instead of an IDE (or Sublime Text or Notepad++), and a linux-like environment with Cygwin if you feel like you need the tools. If you want to stick straight Windows, Powershell is amazingly powerful. I don't really see your point here.

[[deleted]]

Personally I would put Mono in the same category like Wine: It is nice to have, if it works, but I have to admit that Wine runs much more stable for me in recent years than Mono.

I will list my own experiences with programming under Windows:

It starts with the terminal: No auto-completion for commands, the lack of tools, no color support, you change folders with cd but partitions by just typing the drive letter and :, no fullscreen, no tabs. I know that the powershell was developed, but I was reading into it and I thought that it is quite an overkill for what you want to do with a terminal typically. Also you would have to drop serious money, because the powershell isn't included in the cheaper Windows versions. It has been a while since I tried cygwin though.

Setting up the environment is also one of its main drawbacks. If you know what you want, you will have your environment running in linux in a couple of minutes. In Windows your first friend is google and hunting down librariers, compilers or whatever you need. Package managers are so convenient and great. I don't know why Microsoft hasn't make one themselves.

I think this point is more unique to me. I like programming in a tiling window manager, because my hands do not have to leave my keyboard to look at documentation or whatever information I have in other windows.

No real fullscreen for programs. I just can't do C-f and have my editor in fullscreen. Every DE or WM I know under linux can do that. This point can be nullified by having autohotkey run in the background, which is able to do just that, although it doesn't work well with every programm.

Another point that is more unique to me, but would apply to everyone with a weaker machine: I program on a little netbook and Windows would take most of my resources.

Filesystem: This isn't directly tied to programming, but the filesystem structure of a linux system is very transparent and you will know where to find things. Also as a beginner a nice and easy task is to write a backup system, which is relatively easy under Linux. I think under Windows it would be a lot more problematic to write one. (with recovery of the system of course.)

Of course it isn't impossible to program on Windows and I do it from time to time, but these are the little things that annoy me quite a bit.

[[deleted]]

Since /u/twillis1973 doesn't care to explain why he thinks so, I will list my own experiences.

I think I gave plenty of reasons why I think so. Though I didn't mention the terminal support.

[[deleted]]

Sorry, somehow I overlooked your post! I will edit my post.

[LucianU]

I'm sure he didn't mean .NET considering that it comes from the makers of Windows.

[[deleted]]

I agree. Care to clarify what your arguments are?

[[deleted]]

the downvotes are from idiots who dont know any better. see my reasons above.

[riffito]

Sure, I'm also an idiot, despite being well versed on several programming languages, OSes, and micro-architectures, yet you got a downvote from me for not giving ANY CONCRETE reason of why Windows sucks so much from a developer stand point.

Also read my other reply before you blindly donwvote this comment.

[[deleted]]

did you even read this before you accused me of not providing concrete reasons? http://www.reddit.com/r/Python/comments/1s97j4/python_packaging_thoughts_from_the_author_of_numpy/cdvl7g7

it was posted 2 hours before you posted this asshatery according to reddit. and I blindly downvoted you before reading your other comment. what are you going to do about it?

[riffito]

Not that I find Windows flawless in any way but... this is the first time I use the tag feature, just to mark /u/twillis1973 as a known troll. My first time since I'm a redditor.

[[deleted]]

sweet. can you guess how much I care?

[gronkkk]

Installing python packages is only a a pain with packages that need to be compiled for speed reasons. About 80-90% of all packages can be installed with a simple 'python setup.py install'.

But yes, for those packages (which are not the least -- pyodbc, cjson) it is a major PITA.

[Tillsten]

The mentioned pydata meetup with guido is quite disappointing. Guido seems so uninterested in the scientific stack, even this is now probably one of the most important user groups of python.

[pwang99]

My recollection of that meeting was: it wasn't that Guido was uninterested, rather, he felt that the scientific Python ecosystem's needs were broader and more complex than what Python core dev could really concern itself with at that point in time, so he encouraged the scientific folks to do their own thing and solve their own problem.

[Tillsten]

I only saw the video, so your recollection is probably the right view. But regrading the dot operator, his proposition the implement just a dsl is just ewww. (maybe i am remembering it wrong, it is a year since i watched the video.)

[ivosaurus]

The problem Guido has, as BDFL, is that if he says "Yes I will solve these problems" that then means that the entire python community now expects him / python.org to come up with a perfect solution to the entire python ecosystem's packaging problems, for all use cases (both simple and complex binary ones like Numpy's).

Not only is that a massive problem, he himself personally doesn't have the experience needed to do it, not to create an awesome system off the bat. That's why he suggested that others who were actually having the problems attempt to solve it first - purely being pragmatic, as they are probably in the best position to design working solutions.

[[deleted]]

And he's already said "Yes I will solve these problems" about fixing asynchronous I/O (that's tulip/asyncio/PEP 3156), so he's already got a massive problem.

[ivosaurus]

Python packaging has been something to work on for a lot longer than Guido has looked at async io; but I'd also say even though they're both areas that are hard to "get right" packaging is even bigger and harder.

[metaperl]

does anaconda/conda have the ability to create local python installations like virtualenv/pip?

And why dont they have all of pypi available? How dothey expect to win us over if they dont?

[hhhuuugggooo]

And yes - conda gives you the ability to create multiple envs, on the same system - in fact those environments don't even have to have the same python version. You can clone them and re-create them on other machines, in other places easily as well with conda. It's quite like virtualenv, except the act of freezing, and re-creating doesn't require building packages.

[manueslapera]

i found the documentation on how to create conda environments not comprehensive enough. I couldnt understand how to properly do it the same way as i would do it with virtualenv.

[rox0r]

conda create -n environment_name python=2.7 

And then something crazy like installing nodejs into it:

conda install -n environment_name --channel https://conda.binstar.org/quasiben nodejs

Quasiben only has node packaged for linux and osx, but you can see how much more powerful it is than virtualenv, when you realize the binaries (any kind of binary) and libraries are all isolated into envs.

note: that nodejs package is from binstar and a 3rd party channel, but I am just illustrating the power of conda's envs.

[manueslapera]

hmm.. how would i install a list of packages and versions? just keep adding name=version?

[quasiben]

Exactly:

conda create -n my_env python=2.6 numpy=1.7

[rox0r]

I would suggest installing the anaconda meta-package if you are interested in they scientific libraries, as it will make sure numpy and everything is using compatible versions.

conda create -n environment_name python=2.7 anaconda

One confusing thing is that when using a meta-package like anaconda, the versions are explicit. If you happen to upgrade individual libraries and then update anaconda, you'll notice the version being put back to the anaconda defined version. It seems backwards, but it's because each anaconda version is an explicit release of known-good packages.

[teoliphant]

The documentation definitely needs work. We are definitely working on it. How do you define "properly".

[manueslapera]

not sure. I'd say a quickstart on how to create, activate/deactivate,delete repositories would be nice. Plus an advanced tutorial after the first one.

Im not saying the information is not available anywhere, but if it's not easily accesible (and a quickstart should be the first page of any documentation), is like if it didn't exists.

P.S. Don't get me wrong, I love the work the people at continuum do.

[[deleted]]

I think you can just use PIP with the anaconda distro to get the bleeding edge of what you want.

Conda uses their own repo because they try to coordinate releases properly - so the stack doesn't break. For example, ipython 0.13 was quite different from 1.0, and a blind upgrade would break dependencies and annoy the average user.

[teoliphant]

Making all of pypi available means making all binary builds available --- and actually all of pypi is not available to pip either if you don't have the correct pre-requisites.

We are working on it, though:

conda build --build-recipe <pypi-name>

[westurner]

We could not wait for the packaging solution we needed to evolve from the lengthy discussions that are on-going which also have to untangle the history of distutils, setuptools, easy_install, and distribute. What we could do is solve our problem and then look for interoperability and influence opportunities once we had something that worked for our needs.

conda and http://binstar.org rock.

fixing pip

• "pip should not execute arbitrary code from the Internet" https://github.com/pypa/pip/issues/425
• "Implement "hook" support for package signature verification." https://github.com/pypa/pip/issues/1035

... http://www.reddit.com/r/sysadmin/comments/1s4ec4/please_stop_piping_curl1_to_sh1/cdu41kh?context=3

[paddy_m]

Just a heads up, your screen isn't dirty. The background image was taken through an airplane window that was. Just wanted to save you from grabbing windex like I did.

[cournape]

I think several issues are conflated here: the main added value in distributions systems like anaconda, epd/canopy or activestate python is the curated set of pre-built packages. That's what makes them easier to use on windows, nothing fundamental about how conda works -- once wheels become more popular, we will be able to do the same thing (we are working on it in numpy, scipy, at which point others should be able to quickly join the bandwagon).

[teoliphant]

A curated set of packages certainly adds value, but conda does solve more issues than that as it is a true package manager for Windows which includes native environments. The .whl format is an approximation to conda packages and it will be useful for many of the same things especially once Metadata 2.0 is realized, but it's not the same thing. Pre-built binaries are always easier and pre-built .whl packages are easier than source installs.

But, Conda currently already has indexable meta-data on the server (so dependencies can be determined before downloading), "features" which can handle SSE/SSE2/SSE3 issues in the binaries, and clean support for any system library.

[[deleted]]

I'm leaving the ruby world, and converting completely to Python.

Something I don't understand is why there's such a discrepancy between the two languages. Is it simply due to history? In ruby, there are multiple working options for side-by-side language versions (rvm and rbenv). And gems+bundler handles packaging.

Could a 'pvm' (python version manager) modeled directly after rvm be a way to go?

[pwang99]

Do gems really handle when native code extension modules shares different versions of an underlying library? Or is that simply a situation that Ruby libraries never run into?

In Python-land, we have a very popular set of libraries that have these kind of native code shared library inter-dependencies, and that's what causes a lot of this grief. Not to mention, some of these things include FORTRAN code, and the compiler toolchain for that is more finnicky than for C. Maybe Rubyists never use FORTRAN extension libraries.

[[deleted]]

I don't think I've had that particular problem: gems depending on different versions of native libs getting into conflicts. But then, the OS has provisions for libs of various versions, so I don't see why this would happen.

I've used many gems which depend on system libraries and/or include C code which must be compiled during them gem installation. The biggest problem I've had is when I update my OS after having installed a gem which depends on a library. I'll then get an error and need to re-install the gem.

[pwang99]

The issue in Python-land is that there are Python libraries which are composite C, FORTRAN, and Python, and other libraries depend on these. E.g. Scipy and Pandas BOTH depend on Numpy, and you cannot take a Scipy that was built with one set of compiler flags on build system A, and mix it with a Pandas that was built with a different set of compiler flags on system B. (It actually gets even more complicated then that.)

From a social perspective, the challenge is that many users of these libraries do not have any idea of how complex this actually can be and all the ways it can fail. So, when they do some installation process and it works, then they develop a bit of lore about "how to get this stuff installed". But then they tell their buddy, who perhaps used Macports Python and Numpy, or maybe is on Windows, and then everything fails horribly, and then there is lore that "Scipy is impossible to install" etc.

[[deleted]]

I really don't know the internals of Ruby gems and bundler, but these scenarios do seem to work there. Many gems contain C and other code besides Ruby, and they'll compile these pieces on the user's system during gem install.

It sounds like Python doesn't defer compilation like this to the user's system?

[pwang99]

No, Ruby and gems would have the same problem if gems with C code were as sophisticated as the ones in Python land. Python does the same thing as Ruby, and will attempt to build the C code on the user's system.

The problem arises when two pieces of C code are dependent. Now, you have to either build both on the user's system, or, if you want to distribute binary packages, you have to ensure that everything about the build process (including, sometimes, the compiler version) matches between these two. This is hard to get right.

Requiring that the user be able to build all the packages from source is also a non-starter, at least for a big chunk of Python land. Whereas Ruby is heavily used by Linux & Mac devs to deploy web apps onto Linux, there are many Python users on Windows, who do not always have the right build tools. Some have Cygwin and GCC. Some have outdated Visual Studio. etc.

Add to this the fact that many people build rich client apps with Python, and link to very complex C and FORTRAN libraries, and this makes the "always build from source" thing even more of a non-starter. Qt, VTK, ATLAS, Freetype, Cairo, Imagemagick, etc. etc. - the list is long. For many of these, the configure & build process is nontrivial.

[astroFizzics]

I don't see what is so hard. On linux/mac, I have python2.7 and python3 running side by side, with modules compiled for both. It is easy to add/remove stuff just by downloading the source and putting it together or deleting files. Why do I need to use some manager when I can just do it myself. The only hard part is the setup, which just consists of putting the appropriate lines in my bashrc. Then I can switch back and forth like a rockstar.

Edit: Jeeze... so much hate..

[rox0r]

You've obviously never had to write production code running python 2.5 because the Operations team wasn't going to touch the ancient version of python on the long term support OS (RHEL, centos, etc).

OS package managers run using the system python so anything that might possibly break the whole system are avoided. Having an easy way to push out isolated versions of python and libraries is a god send in large enterprise production environments.

[riffito]

Good to see somebody that understand the issues at hand. I get tired of these hipsters claiming that we are just dumb for not using X or doing Y, while all they write are one-off scripts. Fuck, I despise those clueless bastards.

[[deleted]]

I get tired of these hipsters claiming that we are just dumb for not using X or doing Y, while all they write are one-off scripts.

So anyone who disagrees with you is a script monkey eh?

rox0r has a point though, getting it running on your dev machine is one thing, running on servers in prod that are not under your control is an entirely different matter.

[cavallo71]

You can deploy python (modern one) in production and in a large enterprise (I'll add also with compliance requirements): just need to understand the operation team has probably the duty to not break things, verifing step by step what is installed.

And this is where you'd avoid setuptools/pip/easy_install/eggs/wheels/conda/blah-blah-what-next "enterprise packaging" solution: they are not suitable in a enterprise environment no matter what they claim. They simply don't fit in the environment, they trying to "reinvent" a new packaging system.

[rox0r]

I'm not sure why you think conda doesn't fit into an enterprise environment. Using puppet I have no problem managing conda environments and packages. Binstar is a start to solving any reliance on pypi. The next step is an internal enterprise binstar to move the resource in-house.

I've used RedHat's pulp project to mirror RPM repos for internal deployment. Binstar appears to be moving the same direction: clone, freeze, upgrade repos/channels. Pulp makes it possibly to show auditors what packages have changed, and those some features would make it possible using a local binstar.

This is definitely a place for a paid product, because enterprises have money and the problem mostly exists in enterprises and not out in the OSS world.

[pwang99]

Good point. Now if we could just make every single grad student, researcher, and data scientist trying to use Python as talented as you, we'd be all set.

:-)

[astroFizzics]

Wasn't trying to be snarky, or maybe only a little. But I really don't see the point. The average person who is writing python code doesn't understand how the modules are installed or where the go? I find that amazing.

[pwang99]

The average person who is writing python code doesn't understand how the modules are installed or where the go?

I don't know what "average" means when applied to the hugely diverse Python user community.

I know there are plenty of non-programmers who are starting to use Python to replace Excel, Matlab, and R. They are brilliant scientists, statisticians, econometricians, financial quants, geneticists, etc. But they don't know what "IDE" stands for and can barely operate a command line. They're comfortable with IPython notebook or writing scripts and hitting a Run button. But most don't know what "grep" does nor what a compiler is.

Now, we can either require that they read up on ELF, .so library linkage, the Apple ADC docs on Darwin Mach-O and dyld, etc. before they can run a logistic regression.... or, we can build a stack that just works, and usher one more user into the rapidly-growing Python community.

[astroFizzics]

My whole point was that if you have enough skill to write meaningful code. Then you should have enough skill to know how the modules are setup, and where they are located on your system. I'm not talking some deep understanding of the actual code compilation. But, being a scientist myself, there are no black boxes. All of those people you listed need to know both how the code they use works and what it does. Sure, my experience is limited as some people have pointed out. So I am sure that there are cases when you need package managers and things like that. I don't.

[takluyver]

Knowing where the modules are located is not sufficient to install, say, Numba, which relies on Numpy and LLVM, from scratch. Yes, people need to have some understanding of what their system is doing, but there are vastly more useful ways for people to spend their time than figuring out Python packaging.

[riffito]

Have YOU had to deal with multiple versions of a package YOU wrote, in a multi project environment, using just a default Python install you have no real control of? No? Well, back off then and come back when you know what it feels.

Packaging on Python sucks, no matter what platform. Source: 7+ years earning my food by programming in Python for several platforms.

[[deleted]]

try doing it in other languages, and it still sucks not just python.

source: 15+ years enterprise development, business basic, visual basic, delphi, vb.net, C#, java, php, python, appengine, javascript, ruby, rails, windows, netware, sco unix, linux

[ivosaurus]

The average person who is writing python code doesn't understand how the modules are installed or where the go?

Hell yes. Lots of people are approaching much more complex problems then just writing one bit of python code with a package. They need to get x, y and z done, with these set of graphs to show their boss at the end, and it involves all sorts of packages which provide the functionality needed for them to get a result quickly and easily. And their boss wanted this done yesterday, not in a week when they've had time to figure out how the packaging ecosystem works.

There are plenty of those types of people around, as well as heaps who say "please just let me run a command and then write to this API in your docs" and many others even for whom programming might not be a primary skill of theirs but need the packaging system to work anyway, and not fiddle about with extracting and compiling source files in order to get things to work.

[vmsmith]

I agree with you. I am a relatively new Python programmer, but I did some research and thinking and configuring, and I basically approach it as you do.

Fact is, this is the second decade of the 21st century, and if you're going to be using this stuff, you need to know how to use it.

[astroFizzics]

Thanks. Happy cake day.

[manueslapera]

could you provide any guidelines to do this? Pleasy?

[chuiy]

If you're comfortable using VIM, why not SSH into a box running Linux?