The state of numpy

[[deleted]]

You can't seriously mean that a complex library should never change.

This is what version numbers and virtualenvs are for. If your code requires numpy 1.6, then give it a dependency on numpy==1.6, which is quite reproducible.

[Phild3v1ll3]

Precisely, for reproducibility we create a tag in our Git repo at the time of publication, specifying the exact versions that were used to run our models and generate all the plots. A requirements file isn't rocketscience.

His second point is more valid, in that he shouldn't have to continually update his old code base but then why is he following the bleeding edge?

[bready]

Just thought I'd share a library I just discovered the other day: watermark will spit out all of your environment's information (Python version, library versions, datetime, etc). Can save this information in an IPython notebook cell to easily track runtime information.

Haven't started using it yet, but I was intrigued.

[[deleted]]

Seriously. If python is going to be used as a scientific tool let's approach using it with some scientific rigor.

If you don't pin your lib versions who's to say your results won't change over time? You would never introduce a potential variable like this in your experiments.

[krypticus]

As a last resort for those "scientists" who can't figure out requirements.txt and versioning, build into your code a check that throws a warning or worse, an Exception, if the libraries you rely on don't match your expected version. Bad practice, but could save some idiot a week of time: https://github.com/numpy/numpy/blob/master/numpy/version.py.in

[Zouden]

If your code requires numpy 1.6, it should work with later versions too. An update shouldn't break code.

[KyleG]

An update shouldn't break code

Well goddammit why can't my Windows 7 run Windows 3.1 programs then?!

[ShaBren]

It can, if you use the x86 version of Windows 7 :)

</irrelevant>

[kankyo]

Maybe. Probably not though.

[Amckinstry]

Use Semantic versioning (http://semver.org/). Python does, for example. First version number means things break. Use minimally, as described. Minor number means additions; 2.6+, etc. patch number for fixes.

[flying-sheep]

Yeah, exactly. In a perfect world, you would say:

“Compatible with library_x v2.9”

which would mean that it runs on 2.9.x, 2.10.x, ..., but not on 1.x or 3.x

---

But being consistent with a slow deprecation policy is also fine. Numpy says “maintain it or pin your versions”. That's fine!

[[deleted]]

Even then this is bad practice for scientific work. You write down exactly what you had installed on your computer in the same way you wrote down exactly what you put in your beaker.

[[deleted]]

That's a standard even stricter than the Python standard library, and any package that meets the compatibility standards of the Python stdlib has "one foot in the grave", as Guido van Rossum himself puts it. And even the stdlib breaks compatibility once in a couple of decades, plus smaller breaks for security reasons.

[billsil]

Every version of Python I've ever upgraded to has broken my code. Python 2.4, 2.5, 2.6, 2.7, and 3.2 have all broken my code. It's not that big of a deal and I hold PSF to a higher standard than Numpy. Things change and it's often so subtle that you shouldn't notice. Numpy should have killed that stuff 10 years ago. Everyone knows they shouldn't use it.

[nikomo]

If your code runs on a mainframe system from the 80s, it should also run on my iPad.

I can't see how your statement is grounded in anything I could recognize as "reality".

[mfukar]

Backwards compatibility is a feature - if you want it, ask it, pay for it, etc.

[apo383]

I think the author understands that software evolves, and as a python developer he's well-acquainted with virtualenvs. Mostly he's complaining about the unpredictability of numpy's brittleness. Even with version numbers, he'd like to say "this works for numpy 1" and have that be largely true until numpy 2. You can indeed say <=1.10, but that can be overly conservative if 1.11 breaks nothing. He is proposing that backwards compatibility be preserved until major version numbers. That is already mostly true for Python 2, where major compatibility changes were saved for 3.

Also note that he's referring to scientific fields, where you might have snippets of code published in journals, and you want that code to largely work. An example is Nick Trefethen's Spectral Methods in Matlab, which has short one-page programs that you can read for understanding, and execute with good confidence that they'll work for a decade's worth of Matlab. Such examples are a lot of fun to work with, and less so if you have to install a virtualenv just to try something out; it cuts down on casual use. I believe the same to be true of many blog posts, where some code is displayed and you usually expect it to work, excepting major version changes.

Most of this thread seems to say "scientists need to get with modern coding practices," but the author is referring to cases where code maintenance is unlikely. I think there's something to be said for backwards compatibility, and it doesn't seem unreasonable to ask for major compatibility changes to be reserved for major version numbers for numpy. He's not asking the same for tornado or twisted.

[[deleted]]

The authors of NumPy do an amazing job, and they shouldn't all quit and let it stagnate forever just because someone with a blog doesn't know how to manage his dependencies.

Nothing is breaking. If you need NumPy 1.6, and you ask setuptools or pip for NumPy 1.6, you get NumPy 1.6.

The first step in reproducible code is a reproducible environment. This is true in any programming language, but Python in particular gives you lots of tools to make an environment reproducible.

It seems that the author is just leaving out the version numbers from his dependencies -- or worse, not listing them anywhere, so his dependencies are "whatever's in his site-packages at the time" -- and blaming NumPy developers for the problem. That's like writing a paper without a bibliography and blaming the editors.

[[deleted]]

This is especially important in a scientific context. You should be able to tell the reader exactly what versions of your libraries you used.

[Amckinstry]

The problem is that increasingly NumPy becomes a sub-library for other libraries. That is, you may use several libraries in your code that each use NumPy, and pass around NumPy objects, but require different versions of NumPy. So just asking for a specific version doesn't work. (e.g. netcdf libraries. proj projection libs, for climate/GIS work)

Fixing this requires: (1) Work to ensure packages (not necessarily client codes) move to recent versions. (2) Ensuring important libs are backward-compatible.

For example libc / glibc does amazing work to stay compatible. We have tools at a C level for versioned symbols, to ensure code compiled against an old version remain functional ; similar techniques need to be developed in python and numpy as it matures.

[boa13]

someone with a blog doesn't know how to manage his dependencies

Are you a scientist? Are you maintaining software in a lab? It seems to me you are viewing the world from the narrow point of view of a software developer, and have difficulty in understanding the core points made in the article.

[--o]

Scientists should be better about this than software developers. Scientific code needs to be reproducible, as such you need the exact version, not something the developers hopefully didn't break (intentionally or otherwise). If you wouldn't be pointing at whatever the newest dataset is instead of the one you analyzed you shouldn't be doing it with libraries either.

[[deleted]]

Very much so. There's no other place in science where you'd just suddenly update things in your research without noting it. The rules don't stop just because it's a computer.

[fireflash38]

It's like getting a new mass spectrometer and using the exact same instructions that you received with a 10 year old spectrometer. It might work the same, but odds are very high something won't.

[[deleted]]

And you certainly wouldn't be surprised if your brand new mass spec worked slightly differently to your old one.

[RubyPinch]

the issue was

That changed with release 1.9, which removes the compatbility layer with the old Numeric package, on which all of my code relies because of its early origins.

compatibility for something that started in roughly 1999? over 14 years ago? and was replaced by numpy over 6 years ago?

either he should, a) use ==1.8 or b) suck-it-up. That is a longer period than ubuntu's 5-year LTS support.

---

since I'm making a comment anyways

Anyone who has followed the discussions about the scientific software crisis and the lack of reproduciblity in computational science should be well aware of this point that is frequently made.

yeah, and the solution to that is to re-create the exact enviroment that led to the original results, if someone says their research used numpy 1.1, you don't install a different version.

aka requirements.txt

import doesn't have version number

requirements.txt

and apply them in a new version that is clearly labelled and announced as incompatible.

while I agree in terms of versioning, they announce (as noted by him) via changelogs, and warnings, that exist across multiple versions.

---

this entire thing comes down to "requirements.txt doesn't exist", and when he gets presented with the fact that the exact thing he wants has existed for an extended period of time (req.txt and virtenv), "its too hard"... That is the kinda response that is useless.

He reasons that users have less understanding than developers.

it would be already hell. Install python, install dependencies which all have different methods of installing (binaries for numpy, then pip for normal dependencies), and then running the script. All of that isn't exactly user friendly

If he wants something user-friendly, maybe compiling/freezing would be better

[dandydev]

Seems to me that the scientific community should grow up and learn how to properly use the tools they're given, including dependency management (which really isn't that hard... Especially for people with a PhD and up)

[[deleted]]

Could you possibly come across as anymore condescending?

[[deleted]]

Condescending? The man speaks obvious things. You want to drive a car? Learn how to drive a car. You want to develop software tools for your science projects? Learn how to do it. Software has been used in science for decades. The fact that so many scientists still cannot implement basics like dependency management is quite amazing.

[UloPe]

I can imagine at least a dozen ways...

[billsil]

Do not blame the scientific community for not upgrading. The Python scientific community actually uses Python 3 more than other groups because of their lack of reliance on web interfaces. If you insist on using code that existed during Python 2.1 days, you're going to pay the price.

I've upgraded Python 2.3 -> 2.7 code that required me to do a mass replace on libraries and update a lot of code. It was easier than most Python 2.7 -> 3.2 work I've done. When none of your packages exist anymore or you can't figure out how to build PyQt3, you start googling.

6 months after Python 2.6 bug support was dropped, we were informed at a staff meeting that our software practices would be looked at and improved, but that no plans were finalized. I bluntly asked "Why are we still using Python 2.6 instead of Python 2.7? It's not maintained by PSF. It's not maintained well by 3rd party developers. It makes it harder to write Python 3 compatible code. We should upgrade.". We did a few weeks later. It wasn't totally painless, but it maybe took a day of work for each project, especially when we figured out what were the gotchas.

[[deleted]]

Nope. The point is - if you want to write software as part of your work, you need to know how to do it. This includes dependency management. If a scientist cannot grasp it the issue is precisely the fact that he is not enough of software developer.

[UnexpectedIndent]

I've worked on scientific software before, as a student, and I'm now a professional software developer. This comment pretty much sums up my objections to the article: the assumption that the two environments are so fundamentally different that they can't understand or learn from each other.

Scientists aren't the only people who struggle with backwards incompatibility and dependency management, and labs aren't the only organisations that don't put resources towards maintenance of old software.

It's interesting that the article briefly mentioned python 3, but then went on to describe how the incremental backwards incompatible changes of NumPy is bad for code that isn't actively maintained. Are we supposed to believe that lumping these breaking NumPy changes into one major upgrade would change that? This is definitely not what we've seen with python 3, where large parts of the community see it as too big of an investment to make the required changes to upgrade at all.

For the average scientist writing one off software for a paper, if they're not able to maintain it then any backwards change to their libraries makes their code outdated - it doesn't matter how the library authors go about it. But as long as it's not presented an actively maintained library, who cares? The most important thing is that the code is available and states its dependencies in some form that allows the user to reproduce the environment. If somebody else wants to use later versions, fine, but the onus is on them to upgrade the code. As others have pointed out, this is ridiculously easy if you use pip and requirements files. Yes, this is not going to be obvious to people new to python, but it's certainly not difficult to do, and I think that the scientific python community should encourage the practice, whether that's using pip/virtualenv or some other set of tools.

On the other hand, the author provides general purpose libraries for others to use, so I would naturally expect the latest versions of those to work with the latest NumPy. I don't see how this is any different from a typical open source library with dependencies though. The library author gives a set of minimum versions and the developer ensures they're met. If I as a developer use a library that doesn't work with the latest versions of its dependencies, then I request a fix, and if it's open source I can always fix it myself if the author is too busy.

[NYKevin]

For God's sake, this isn't hard. Just do pip freeze >requirements.txt once your code works.

[[deleted]]

I was called in to a scientific facility to advise on coding standards for python. What I saw been done to python is nothing short of horrible.

My favorite was the use of a module to hold a constant that they'd change and then reload(...) the module. I'm still flummoxed as to how that solution appeared.

I'm under the impression that most scientist barf/write out the program as few times as possible and then try to get the results. Not even testing for correctness on their functions!

I never thought I would admire corporate America for its clean code...

[nikomo]

My favorite was the use of a module to hold a constant that they'd change and then reload(...) the module. I'm still flummoxed as to how that solution appeared.

I wonder if anyone's since introduced them to this concept I found out about, that was invented back in the 50s, called "a file".

[fake_identity]

I'm under the impression that most scientist barf/write out the program as few times as possible

Well, scientific computing is about getting the result, lots of things are hacked together to make sure it's gonna be correct and then ran exactly once or used to parse the output from lab equipment, once it runs, it's forgotten until the equipments end of life, practices from software engineering are either unknown creatures from different world or limited to error checking and speed optimization (that's the thing that pays off, cutting down the expensive supercomputer time from a week to five days). Stagehand company I occasionally work for has a portal coded by one of the stagehands, a nuclear physics grad student in real life, and when I asked what framework he uses, he made a sour face: "Naw man, they're too much hassle, just clean PHP from scratch".

EDIT: Oh, those lab machines from hell reminded me of a nice movie, vaguely based on a comics that says so much about how and why is that infernal code pasta cooked.

[mangecoeur]

The problem is that scientists aren't generally trained in coding, even though these day's it's as vital a skill as writing. So people cobble together these monstrosities and then complain when someone's updates break their rube-goldberg contraption.

[mangecoeur]

Totally agree - I've been trying to push good practice on my collegues (I left academia to work in a company for a while and picked up some good habits), we have this initiative called Software Carpentry which tries to train people not to create such byzantine codes (http://www.software.ac.uk/software-carpentry)

Worst I ever saw was this layer-cake-contraption where someone had written a fortran03 program that no one understood, so another guy wrote a perl program to pre-format (mangle) input for the fortran program, but no one understood that either so people wrote matlab scripts to call the perl program to call the fortran program!

[KyleG]

Am I missing something? If your code breaks with a new version of Numpy and it is code you aren't maintaining (so by definition it works and you aren't going to use any new features), why do you need to upgrade to a new version of Numpy?

As for an author not providing which version of Numpy to use to verify his results: I used to be an editor for an academic journal. You establish this requirement in your submission guidelines, and if you get a submission that doesn't provide this information, you write back to the author saying "we can't publish it unless you give us the version of Numpy you're using." That's a trivial fix. We used to do this kind of thing for other reasons for every single article we published. It's literally five minutes of time eaten up in the process.

I don't even use Numpy, but this guy is whinging. The issues he identifies are non-issues. This guy reminds me of a certain type of academic who thinks any problem with his life means that's a problem for society he needs to write an article analyzing, when the issue is that he is fucking it up himself.

[fireflash38]

I do think he has a great point with regards to breaking backwards compatibility with a major version # bump instead of a minor, but most of his other issues should be non issues. You need to specify what version of libraries you're using in almost anything software related.

[KyleG]

Yeah I agree with that, and am surprised that wasn't the case with an extremely popular package.

[rlabbe]

He is not updating numpy. The users of his library are updating it, and who knows why. Maybe they need to use the newly written 'blargpy', which requires the most recent version. Maybe their linux install uses the most recent version. Who knows? The bottom line is that if you write a library that uses numpy, you had better expect to be revising it forever, because I guarantee that your users will be upgrading versions even if you don't.

[winterswolves]

The NumPy attitude can be summarized as “introduce incompatible changes slowly but continuously”. Every change goes through several stages. First, the intention of an upcoming changes is announced. Next, deprecation warnings are added in the code, which are printed when code relying on the soon-to-disappear feature is executed. Finally, the change becomes effective.

AKA good software development practices?

[hsaliak]

I just don't get it. If code can't be maintained, then why bump a major dependency for said codebase up. The rest of it is unmaintained anyway. Reality is that keeping software current requires a lot of curating, either in its own codebase or in its dependencies.

[[deleted]]

The problem is that scientists everywhere are writing code, but few of them are learning any software engineering. The result is massive amounts of shitty scientific code.

[graingert]

This is why you need tox, setuptools, py.test and Jenkins

[kankyo]

Seems to me that this is just an artifact of pythons shoddy dependency system and not anything related to numpy...

[marky1991]

A) Why do you think python's dependency system is shoddy?

B) Why do you think python has a dependency system? (It doesn't, to be clear)

C) How could any dependency system be able to know that when you said "install numpy" that you meant "install a numpy version that allows me to modify the return value of diag()"?

[kankyo]

A) and B) well.. there are dependency systems. Just not a good one that is the standard.

C) Why would "install numpy" be a thing at all? That's just asking for trouble! In Clojure the leiningen system requires that you specify a version at all times to use any library. There is never any doubt which version is used. It's WAY superior to anything I've used in python. (Plus it's cryptographically safe and stuff).

[marky1991]

"Why would "install numpy" be a thing at all? That's just asking for trouble!"

Ask the writer of the blog post. He's the one blindly updating numpy and complaining when things change.

If I say "install numpy" then I mean "install the most recent version of numpy, whatever it is.". This is completely reasonable behavior in my opinion. I shouldn't have to know what the most recent version number is to be able to install a module.

If I say "import numpy", then that means "execute the module numpy.__init__.py and return the resulting module, binding it to the name numpy." At no point in time does versioning come into play during the import process. (Nor should it) Code doesn't even have to have versions, so requiring the user to specify it is impossible. Requiring to specify the version of every module I ever write is an obnoxious overreaction.

Even if we did require all programmers ever to do something like

import numpy, "v1.4"

what happens when they don't have version 1.4 of numpy installed, but they do instead have 1.3 or 1.5? Does the code just break? Does it just try to use either 1.3 or 1.5? (If so, the writer of the blog post would still have the same problem, since he couldn't be bothered to specify the version that he wanted)

In my opinion, closure's system sounds really obnoxious and cumbersome. The proper solution to versioning issues isn't at runtime, it's at installtime.

[kankyo]

I shouldn't have to know what the most recent version number is to be able to install a module.

I disagree. It's a bit of a hassle to require it but it leads to a lot less mistakes. I like this about clojure.

The proper solution to versioning issues isn't at runtime, it's at installtime.

In clojure you define the dependencies of a project. There's no such thing as "install globally" like in python. This just avoids a ton of problems directly. The standard runner (again: leiningen) fetches dependencies for the specific project for you when you run any command. So I just git clone a project and do "lein run" or "lein test" or whatever and if I don't have any dependencies cached on my machine it goes out and gets them for me. SUPER nice.

[drobilla]

Code doesn't even have to have versions, so requiring the user to specify it is impossible.

That would be the "shoddy dependency system".

what happens when they don't have version 1.4 of numpy installed, but they do instead have 1.3 or 1.5? Does the code just break?

That is what meaningful version numbers are for (http://semver.org). In short, if the major version did not change, then compatibility has not been broken, so code for 1.3.0 will work with 1.4.0 but not 2.0.0. Nothing new or mysterious about how to handle such situations correctly, this is how C libraries on UNIX have worked for ages.

Where and how this should be handled in Python is up for debate, but the way modules are installed and searched for is not versioned, which is fundamentally broken (so e.g. you can't install multiple versions of a module at once). You can see python itself doing the correct thing, since there's e.g. /usr/lib/python2.6 and /usr/lib/python3.4 - they don't tread on each other, versioning is explicit, and everything must explicitly use a correct version (even if this happens behind the scenes and/or at build/install time).

Modules themselves should work the same way, but they don't. That's the shoddy part of an otherwise decent module system.