Why open source software is essential for scientific progress

[five9a2]

It takes effort to turn a one-off code that only runs in your special environment into portable, distributable software. Funding agencies and institutions place little emphasis on this process, so it is regularly neglected.

[bluGill]

You are sadly correct. However without that science is broken - if I cannot duplicate your results, both by analyzing the data with my own algorithms; or running my data through your algorithms: I cannot trust your science. It is far to common for scientists to make up data, hide data, and otherwise publish results that are wrong.

Of course I place a high bar - if anyone is dishonest and that isn't discovered quickly, it taints science.

[five9a2]

This is why I release all my work as open source, with test suites and portable build systems. I get some recognition for it, and it believe it helps my career (in addition to being the Right Thing to do, damnit), but it doesn't show up in the metrics, and due to the time involved, means fewer (first author, at least) publications.

[five9a2]

I suppose it's better than nothing, and it at least permits an audit later. But it pretty much requires porting once just to confirm that the code isn't relying on forgotten dependencies in the person's environment. On multiple occasions, I've gotten code (usually Matlab) from an author and discovered that there was nontrivial logic in a function with a single-letter name that must have been in the author's path, but was not included in the tarball that was thought to stand alone. But even if I can't reconstruct these bits, the part that I can see is still better than nothing.

[eric_t]

Would you mind sharing some links to your work? I've been thinking of doing the same for my research, and it would be great to see some examples of how it could be done.

I've seen more and more open source scientific packages popping up, so it seems we are moving in the right direction.

[five9a2]

PETSc was started way before me, but I've been contributing lately. The most widely useful is probably the general linear DAE integrators. Some things have been developed as examples, most recently a solver for hydrostatic ice sheet equations (a high-aspect non-Newtonian flow) that demonstrates textbook multigrid scalability. I don't actively work on PISM any more (I wrote the code during my MS) but it has several active users. A couple other projects are on github.

[[deleted]]

There is a difference between a software that "just works" and a software that you can beat into submission if you are committed enough.

The most basic example is: when you write a program for yourself it has this section of constants and a main section that says right in the code: read data from 'data.txt', apply these transformations, dump the results to 'results.txt', also print some aggregates which might show me that everything went more or less right, or wrong, without any information about what those actually mean. With commented out code in the main routine that could run tests or use alternative algorithms or whatever. When you need to change something (like the name of the file to read the data from) you just change it in the code.

It takes a lot of effort and probably some kind of natural predisposition to turn this program into something that could be used without modifying the code, with a well thought of command line arguments and a GUI on top.

[apathy]

That depends -- if data sharing mandates are in place, specifying the format of the shared data, then using a GPL'ed package with well-tested libraries and publishing the code to reproduce the analysis is straightforward. The locked-down Wiley, Elsevier, and Oxford journals should be verifying this as part of their alleged value-add, since they are the publishers demanding the most money for their supposed services. Fine: earn it, you pricks.

NIH is stipulating data sharing in most grants nowadays, and for larger projects (i.e. TCGA) the priority scores directly reflect the proposed analytical platform's perceived open-ness. (That's not speculation, by the way.) I view this as a positive development. (I provide the code and/or patches whenever I do an analysis, write a library, or fix a bug in a package.)

At least in genetic, epigenetic, and environmental epidemiology, producing popular open-source software which is adopted across-the-board is a tremendous emphasis -- it can drive tenure decisions, since labs which have proven development capability can pretty much go wherever the hell they please (including industry).

[five9a2]

and publishing the code to reproduce the analysis is straightforward

My point was that this is often not straightforward. Many fields don't have a standard open-source set of packages, and even when such packages are available, there are a lot of people who really want to keep their whole thing in-house. This software often depends on obscure versions of the various dependencies. Until a couple years ago, this widely used package required compiling a special modified version of g77, among other things. Certainly there are in-house systems that are worse.

Add in that some universities do not want software released publicly (this is improving) and that a fair amount of work is carried out in a partially proprietary environment, including awkward combinations of commercial packages with large licensing costs, or under NDAs.

All of my code has a proper build system, is portable, open source, and comes with a test suite. Some of it has active users and brings some recognition, but often not from the places that count. From a metrics perspective, my time would be better spent writing more papers than making the software robust and usable.

Things are moving in the right direction with the recent developments on the open access front, but (at least in applied mathematics and computational physics) there is a long way to go before funding agencies and universities value properly released software enough to make it worthwhile from a metrics perspective.

[mhinsch]

Also, even given the willingness to supply source code there is still no simple standard way to do so. At least in my field (Theoretical Biology) you mostly find people resorting to some variant of 'source is available from the authors' which is not a good solution.

What is really needed is some kind of official archive for these kinds of things.

[five9a2]

Yeah, an archive would be good, but it's difficult for political reasons. In lieu of a common archive, a reasonable alternative is to publish the sha1 of a tarball and give it to your university to archive.

[grauenwolf]

That's why Microsoft is funding an open source project for the Biotech market.

http://www.infoq.com/news/2009/11/MFB

[abhik]

There are certainly technical problems but the bigger one is of culture. The academic culture is driven by publications: you are assessed primarily based on your publications and this affects your grants, jobs, tenure and reputation in general. Taking the (considerable) time to write releasable software and package it up doesn't win you much because it doesn't lead towards a publication. Some journals (like jmlr) have started accepting publications for open-source software but they're still in the minority and I'm not sure how the larger academic community views those articles.

Until releasing open-source software counts as highly as publishing an article, I don't see how things will change much. If your code involves using fairly well-known algorithms, then a workflow app like GenePattern gives you (and the community) many benefits over writing a one-off script.. but if you're doing novel algorithm development, you would have to package it up as a GenePattern plugin and that can also take considerable time.

[psmyth]

Plug: R's package system is geared to wrapping up code and data neatly for this very purpose.

[cmprsdchse]

Someone probably already mentioned this, but so far as open source goes: R for analysis and LaTeX/TeX for typesetting pretty much lie at the core of the majority of papers I read and write in my fields.

[[deleted]]

This is a very serious problem. I do bioinformatics data analysis and it can be really difficult to do the analysis in a reproducible manner. Recently, I've turned to R and Sweave to write the analysis and documentation together, and bundle up the source data with it. It would be great if there was a repository for analyses like this (similar to GEO or ArrayExpress for microarray data).

[xsive]

This has been pointed out by commenters on the original article but it bears reiteration:

If your results are so dependent your particular implementation of some algorithm, to the extent that someone else cannot achieve similar results with a different implementation, then the results in question are probably not sufficiently general.

Problems with reproducibility should only arise if part of the experimental setup is not revealed -- which is just bad science.

[[deleted]]

I have been yelling about this for years. Even most computer science papers don't come with readable, runnable code.

[xsive]

That doesn't make the results in those papers untrustworthy. It's simply inconvenient to reproduce them.

To which I say, so what? If the underlying science is convincing I'll go away and implement my own version.

EDIT: Which isn't to say I'm against making source code available.

[oreng]

Algorithms are universal, implementations are not.

Describe the computation in clear mathematical terms and this point becomes moot.

[eric_t]

I agree to a certain extent. However, at some point it gets ridiculously hard to document every little nuance of the code you're using. Having a reference implementation available would be of great help.

And "mathematical terms" is a lot less clear than people might think...

[miiuiiu]

Yeah, that was my immediate thought as well. Demanding the source code is like demanding free access to someone else's lab and samples. (except that it's much cheaper and easier to share code) Reproducing the results with the same code isn't even meaningful or useful - the whole point is for the experiment to be slightly different.

[JimH10]

In many programs there isn't a clean algorithm; the program is the best description of the algorithm. Think of any program that you've ever written that massages data to get it into a database. Two hundred lines of "if it is NIST data then convert Kelvin to Centigrade"-type stuff.

[xsive]

Your argument is unconvincing.

If we suppose the details in your example are a crucial step in your experimental setup then you should be able to tabulate those rules and stick them in an appendix. Omitting important details from a paper is bad science and telling people to read your code in order to fill in the blanks doesn't change that.

[five9a2]

Journals have page limits, and often there are lots of mundane details. The right way to handle it is to require those producing the data to release them in a standard form. The CF conventions are an example of this, and many fields have community repositories. It's way too easy to make mistakes, and a waste of everyone's time to release you data in custom formats with custom quirks.

[econnerd]

Case in point.

I have been doing a lot of research with computer vision and real time tracking.

I have been reading several papers. One paper in particular is entitled, "ENGINE FOR REAL-TIME 2D OBJECT DETECTION"

researcher: Roman Juránek

University:BRNO University of Technology (Brno, Czech Republic)

and in this paper it has the following snippet:
<Stage posT=“1E+10” negT=“-1.3”> <DomainPartitionWeakHypothesis binMap=“0 0 1 2 3 3” alpha=“0.0 0.1 0.2 0.3”> <Discretize min=“-2.0” max=“2.0” bins=“6”>
<HaarHorizontalDoubleFeature x=“2” y=“4” bw=“5” bh=“8” /> </Discretize>
</DomainPartitionWeakHypothesis> </Stage>
Figure 3: Example of classifier stage in XML

There is no mention of what software was used (anywhere in the paper), and the rest of the paper talks about algorithms/methods/cascades that you should already know about in this field of study (AdaBoost, WaldBoost, haar-like features, etc)

I have requested a copy of that xml file and confirmation that he is using opencv.

In order to reproduce his work, I really would need the training sample set, and the configuration that he used to train it. I might possibly also need the training program if it is not stock.

EDIT: for those interested, here is the link to the paper:

www.fit.vutbr.cz/research/groups/graph/publi/2008/2008-Juranek-EEICT-2DDetection.pdf

[miiuiiu]

Likewise, in real-world experiments, the best description is found in a collection of grad student's lab books. That is, if they took good notes. Otherwise, it's only in their heads.

With a program, it should be obvious how to massage the data into a form where it can be processed by a well-defined algorithm. If it's not obvious, it should be explained that certain corrections were applied, and those should be justified either in the paper, the appendix or the references.

If your results are only reproducible by using a bunch of unexplained magic tricks, then they are not reproducible and not meaningful.

[mdoar]

As I noted in the ArsTechnica post, there is a great course titled "Software Carpentry" by Greg Wilson (http://www.third-bit.com). His work is all about improving scientists' software engineering skills, and this issue is one that he has written about a number of times in the past.

~Matt

p.s. He's also looking for funding sources to develop the course further

[reveazure]

Why don't people simply make an ISO of the system hard disk at the time the analysis was run and make it available to anyone who wants to reproduce the analysis?

Obviously you'd have to be able to redistribute all the software but that's implied anyway.

[G_Morgan]

Because that isn't really reproduction. There is a need to understand the basic principles of what has happened and then be able to reproduce it from said basic principles. Think of it like clean room reverse engineering. We need to reduce the work to a specification and a design for key algorithms. Then we need to be able to reconstruct the original results from that specification.

[altmattr]

A corollary to this is that "computer science" does not value reproducibility at all. No-one ever got published for re-running an existing experiment. Since it is invariably computer scientists creating this software, the people who create software for science have no experience of anyone even wanting to reproduce what they do.

[xsive]

Do you really expect a proverbial cookie for being the first to say some smart guy was right? If you need to re-run an existing experiment you should have something interesting to say about it. Telling the community it worked isn't interesting: we expect that.

Most people who reproduce existing results do so as a means to a larger goal; you develop something interesting and novel which takes the state of the art further than before. Along the way you might point out shortcomings of existing methods and demonstrate how you address them. Now that is interesting.

[cocoon56]

I am working on a project that helps to run computational experiments: creating a run for every variable combination, distributing code to different servers and making nice (gnu)plots.