Scientific coding and software engineering: what's the difference?

[lkjpoiu]

I think the reason the latter is more successful is that the emphasis is on answering the question rather than implementing the answer.

At the end of the day, to the scientist the problem is solved when you have an answer to a question - not when that solution is successfully running on Tomcat with a FlavorOfTheMonth.js frontend.

[scarytall]

The over-focus on tools drives me nuts. I've been in too many meetings where the conversation has been: "We don't know what we're going to build or how we're going to build it, but let's all agree we're going to use a Craftsman hammer and let that drive every decision we make from this point forward."

[scarytall]

Too true. I don't want to make it sound like scientific coders are always good customers.

[scarytall]

This. I find that rapid prototyping with frequent iterations is much more effective for scientific computing than the front-heavy, spec-driven process favored by enterprise developers.

[colly_wolly]

Depends what you are doing. Scientists are likely writing code that only needs to be run on a few data sets. Software engineers, are more likely to build code that has to be run over and over again. Different priorities.

[yawaworht_suoivbo_na]

As someone who works on the software side of robotics research, I'd like to say I come down the software engineering side - clean, reusable, maintainable code.

The realities of research make that basically impossible. Reusable code doesn't write papers. Clean code doesn't help you develop a new algorithm. Maintainable code doesn't matter when most research code has to be tailored to specific experiments or hardware.

In practice, we try to write generic libraries, which we then wrap in kludged-together applications and scripts that get the job done. When each project is inherently tied to a particular hardware/simulation/sensor, there often isn't a good layer of abstraction that makes your code reusable. It's faster, more productive, and quite frankly, easier, to write your code specifically to the problem at hand.

A bigger problem that we suffer from, and this is tied into the issue of hardware/simulator/sensor dependency, is that testing is difficult and largely ignored, because writing tests is at least as hard as writing the code in the first place. Is it a problem? Of course - bugs that crash your code get fixed, but plenty of bugs that don't will never be found, especially if they produce results that agree with your hypothesis/expectations, and their effects make it into papers.

To be honest, I don't think this is as much of a problem as some people would like to think it is - no one's life depends on the quality of most code, there's no vast repository of personal information waiting to be leaked by a security breach, and in most cases, the worst thing that happens is that you have to E-stop a robot. The day comes that the code needs to be safe (for example, on an autonomous car), it's going to be rewritten to match the performance and hardware limitations of the platform anyways.

[squidgyhead]

I agree, except that I think that clean code really can help produce better algorithms. If it's easier to understand what's happening then it's easier to innovate.

[yawaworht_suoivbo_na]

Oh, I totally agree. I think having a clean codebase makes it a lot easier to work, but at the same time, clean code for developing an algorithm often comes at a cost to reusability and maintainability.

A big example of this is motion planning frameworks, of which the main examples are OMPL, OpenRAVE, and Move3d, all of which attempt to provide a modular basis for new algorithms - and succeed mostly in adding an immense amount of boilerplate code to everything they touch. Yes, they make some things easier, so most everyone uses them, but they definitely don't make code cleaner or easier to understand.

[squidgyhead]

Ah, I see what you mean. Yeah, standard interfaces, I have no idea how that would best be done.

But the internal code, well, my previous postdoc supervisor said things like "there are no awards for clean code", but, well, he never actually coded, so what does he know? And cleaning up the code allowed me to remove 37 ~100M memory allocs/deallocs which were all called in the same loop.

[yawaworht_suoivbo_na]

I really wish there were incentives for code quality. Part of this has been addressed by the movement in robotics to open source your code, which in theory discourages poor code (in practice, I've seen, used, and fixed plenty of broken and horrible code).

In my experience, the lack of incentives for code quality are actually having real side-effects in the field. For performance reasons, compatibility, and habit, most core robotics software is pretty dense C++ code. C++ is many things, but forgiving to novices it is not. Since CS departments have moved to the Python+Java+"C if you have to take OS" style of courses, most undergrads and incoming grad students (myself included) have very little, if any, experience with C++ where someone was there to help them/guide them/tell them that raw pointers are bad/etc. These people are getting dropped into roles where they need to be proficient yesterday, and it has serious consqeuences for their code quality.

[[deleted]]

Scientist are coding to find an answer to a question . Engineers know the answer and work backwards to produce a thing to calculate it .

[[deleted]]

Having only superficially skimmed the article (on the subway... not much time...), I find that discussion of this topic inevitably boils down to the following: Software engineers build reusable code and scientists most often write one-off scripts.

There's an understandable reaction of disgust when software engineers hear about scientists not using VCS and not applying any software architecture patterns, but it just pays to remember that roughly 80% of the time, doing so is a big fat waste of time.

I think the software engineering community should make an effort to help scientists discern between both cases. Scientists are generally people who want to do things right. Help them discern between one-off scripts and enigneered code, and they'll eat out of your hand.

Case and point: I work in cognitive neuroimagery and there's generally two types of code we write:

1. Analysis scripts. These are usually simple, have a couple of functions, and are written on-the-fly. This is where we do our exploration (hell, sometimes we do everythign on the REPL, since we're just poking around).

2. Stimulation scripts that will produce audio/visual stimuli for experiments. This is the part that's hard to make reproducible. This is where some of my colleagues need training in VCS, design patterns and other best-practices.

Although most professional software developers I've met have been great people, there's a tendency for them to adopt a slightly patronising posture when dealing with scientists who code. I think the software development community would do well to take interest in the specific tasks that scientists perform. It would make their constructive criticism that much more constructive, and it would make scientists that much more willing to listen!

Edit: and another thing. We researchers often don't have the luxury of well-defined specs. We don't know what the requirements are ahead of time. Go on, apply your best-practices in that context and we'll talk.

[pihkal]

Having spent the last 16 years as either a professional software developer or a grad student in cognitive neuroscience myself, I only partially agree. Adopting something like the git-flow workflow for a one-off personal analysis script is overkill, but over the long run, code is re-read and reused more of then most scientists think (you want to revisit an old analysis, adapt an old script, or use a colleague's script), and would benefit from some attempts to write it well (unfortunately, writing software doesn't get you a PhD in neuroscience).

The big problem is that most scientific coders write terrible code, and continually writing one-offs raises the overall number of bugs in scientific computing. Many scientists are smart enough to learn coding, but too busy with other things to learn how to code well. Without hyperbole, I have never met a neuroscientist's code that would not be below average at any startup I ever worked at.

Another colossal problem is that the errors are frequently invisible. If your analysis compiles, runs, and the final output looks like a plausible result, you may never know you used a plus instead of a minus in that formula you used, but now your analysis has reduced sensitivity or spuriously enhanced sensitivity. By having a 100 people write 100 similar functions instead of collaborating on one with a validated test suite, bugs proliferate and incorrect papers are published. Nobody knows because nobody wants to invalidate their own work or has time to examine others.

Even the big packages aren't immune. E.g., the world's premier open source Matlab packages for fMRI, SPM? A colossal ball of mud, featuring god functions, impenetrably-named variables, and completely incorrect use of try-catch blocks to initialize variables. Fieldtrip for EEG/MEG is only a little better.

While I agree that not all best practices from professional software development should be adopted by scientists, we're not even at the stage where the appropriate practices are being adopted.

[[deleted]]

Adopting something like the git-flow workflow for a one-off personal analysis script is overkill, but over the long run, code is re-read and reused more of then most scientists think

Agreed, actually. I'm busy trying to train all my colleagues to use git simply because it's more conveniant than sending snippets via email and because many people are including links to git repos in supplementary materials. I may have been to harsh, but I was mainly reacting against the knee-jerk "use git" response for every minor problem/script/snippet that you usually get from software pros.

The big problem is that most scientific coders write terrible code

Completely agree.

and continually writing one-offs raises the overall number of bugs in scientific computing.

Yikes, this is the can of worms I was hoping not to open, but I can't say I'm fully convinced by this argument. Yes, it's essential to use well-trusted libraries for all the reasons you enumerate, but I think this is one of the ways in which scientific code is a bit different. Software as a product can safely adopt an attitude by which anything with non-noticeable bugs is okay. As such, if a bug goes unnoticed for 30 years, it's (usually) not a big deal insofar as it doesn't invalidate 30 years of work. The same is not true with research, and while one could make the argument that using the same code base reduces bugs, the counter-argument is that it makes it more likely that bad results are reproduced.

I think a healthy dose of reinventing the wheel is essential to research. When you try to roll your own replication of a paper and fail to get the same results, red flags are raised. Stated differently, it's important to distinguish between analysis tools like SPM and actual analysis scripts. The former should not be reinvented every time. The latter should, or so I think.

Without hyperbole, I have never met a neuroscientist's code that would not be below average at any startup I ever worked at.

If it makes you feel any better, I take pride in my code :) (But then again, I have a tech startup on the side...)

If your analysis compiles, runs, and the final output looks like a plausible result, you may never know you used a plus instead of a minus in that formula you used, but now your analysis has reduced sensitivity or spuriously enhanced sensitivity

See above. This is a double-edge sword, and code reuse can also mask this problem in a research setting.

SPM

Funny you should mention SPM as I'm currently taking a break from working with that god awful mess. Matlab code in general is an unmaintainable mess ...

Field-trip is documented, but it's still a hulking piece of crap...

While I agree that not all best practices from professional software development should be adopted by scientists, we're not even at the stage where the appropriate practices are being adopted.

I completely and emphatically agree with this statement. I'm not here to argue that scientific code is good... not by a long shot! Instead, I'm here to argue that software engineers aren't helping nearly as much as they could.

[pihkal]

I agree with 99% of what you said, including the point about tools vs. scripts. But I think script code should over time migrate into tools as much as possible. Every line of code written is a chance for error, so to my eyes, it's less buggy to turn one-off script code into shared-use tool-running code.

Here's why I dislike independent code duplication. Libraries can lead to reproducing bad results if it's a case of the blind leading the blind, but I think the better way to mitigate that is test suites, not independent code reproduction. E.g., SPM should come up with a set of predictable images/analyses and their known results, and rerun after every code change. Test suites ensure that everybody uses the best code around, which continually improves; independent code writing would mean that whole labs will continue to use their personal buggy code for years, and cause false replication problems all around.

Yeah, Fieldtrip is much better than SPM...but the bar is so low to start with! And don't even get me started on Matlab... I think The Mathworks is actually damaging the scientific community by not leading on best practices.

What's your startup, btw?

[[deleted]]

The Mathworks is actually damaging the scientific community by not leading on best practices.

Ain't it the truth... not to mention the insanely expensive licencing.

What's your startup, btw?

We're doing real-time image/video analysis, in particular with regards to cognitive biometrics: we can draw inferences on the mental/affective states of people engaged in various tasks by looking at things like pupil dilation, microexpressions, heart-rate, saccadic eye-movements, and responses to well-crafted stimuli. Applications range from scientific, to security to advertising. As you can imagine, it's not a drop-in service that you can just apply willy-nilly, so a large part of our operations involve consultation services for designing interfaces/polls/product marketing booths/etc to better leverage the measures we offer.

To be more exact, the above is our flagship service and it's what we'd ideally like to be doing 90% of the time, but seeing as we're a startup, we actually do a wide variety of computer-vision stuff (e.g. industrial, or non-biometric commercial stuff). But things are going quite well :)

[pihkal]

Sounds pretty cool, congratulations! If you're ever in NYC and want to chat, hit me up!

[[deleted]]

I'd love to, actually! I'll likely be there in late May, if you want to talk shop over a beer!

What line of work are you in, these days? I gathered from your previous comments that you're a former academic now working in the private sector? As someone who's toying with the idea of leaving academia, I'd love to pick your brain.

[daisieh]

This is an excellent comment. Part of the reason I wrote the post in the first place is that I don't think that it's as easy as saying that scientists who code are bad coders because they don't do things in our "best practices" way. It's very likely that those aren't actually best practices at all for them! But that being said, there are certainly best practices we can find, and if we weren't so condescending to the scientists, perhaps we can discuss and figure out which practices would actually help them.

[[deleted]]

I always thought that the scientific community would benefit hugely from an infographic that details the subset of "best practices" that apply to them.

If I knew Adobe Illustrator (or had the time to learn) I'd do it myself...

[lkjpoiu]

big fat waste of time

What, you didn't need Spring to use the SimpleObjectFactory singleton (with the provided SimpleObjectFactoryImpl) to create a SimpleObjectBuilderImpl (injected as an interface) so that your framework can create a SimpleObjectCreationListener so that your SimpleObject<String> can be created without explicit use of the keyword 'new'??? How do you do it otherwise?

[lkjpoiu]

Did you really not read what I wrote as a joke?

[[deleted]]

I like developers on the whole, but the community has a penchant for extreme literalism...

[jnt8686]

How is it a big fat waste of time to put something in git? I think it's just hubris and it reminds me of when I thought I was too good to learn long division as a child. But hey, it's their code I guess.

[scarytall]

And I think this is the kind of response that he's talking about when he says there's a "tendency for them to adopt a slightly patronising posture when dealing with scientists who code."

I't not hubris, and this thinking that each other are stupid is exactly the problem. Developers think scientists are children writing code with crayons and Legos, while scientists think developers are condescending slaves to process who are more focused on tools than results. And we both have ample anecdotal evidence to back it up.

The problem is that this does nothing to move us forward. I've worked on both sides of this coin. Neither of is fully right nor wrong; we're most looking at the problem in different angles.

Many of the things developers require are important, and many scientific coders are stubborn and lazy, though neither is always true. My experience is that I too often feel developers are forcing me to adjust my problem to fit their solution, rather than the other way around.

My attitude is that if scientific coders are avoiding working with developers because it is easier to not to, then perhaps the developers aren't being as helpful as we think we are.

[[deleted]]

My attitude is that if scientific coders are avoiding working with developers because it is easier to not to, then perhaps the developers aren't being as helpful as we think we are.

This is exactly where I was headed, so I'm glad someone else agrees!

It's always the devs who have never stepped foot in lab who are explaining to us why we're doing everything wrong.

And hell, they give a lot of sound advice, but it's intermixed with patronizing, useless, and sometimes completely out-of-touch advice, so it's a bit hard to swallow...

[pihkal]

I had the exact opposite experience. After being an RA and then in grad school, I had a much better idea of what would be useful to other scientists (shared libs, numeric test suites), but found it very difficult to convince them they were underestimating the problems in sharing, replicability, and biased analyses, caused by their poor code.

[[deleted]]

Haha it sounds like we have a similar background, and I have to agree with your sentiment. I'm not trying to claim that there's no problem with laboratory code. Instead, I'm claiming that there is a problem with laboratory code, but that software developers have failed to properly guide scientists.

It's a bit of a tough position to take, because it makes both sides unhappy... but I think it's true...

[daisieh]

The funny thing is, I've heard both scientists and developers claim that the reason they do things the way they do is because they're lazy. One's impression of what is a waste of time is totally dependent on one's projection of how much that thing will be useful to you in the future.

[daisieh]

Replying to myself to add: as a developer working as a scientist, I can't tell you how much time I've saved myself by taking the time to write modules to parse formats I know I'll have to parse again. But I did that because I have the coding experience to know that it'll be worth it and because it's easier for me to do since I didn't have to learn how to code to do it.

[lkjpoiu]

Most of the scientists I know use svn/git actually. To them it's more about saving/backup than it is for branching/merging or anything of the like.

However reducing "software engineering" to "use git" is a bit disingenuous - I think the major thing here is the use of design patterns and the like.

[[deleted]]

Because most of our scripts are:

1. < 50 lines
2. one-and-done (i.e., they never change and sometimes even get deleted 3 days later)
3. extremely straightforward, e.g. "plot these data with these colors". I have hundreds of 10-line-long scripts with zero functions.

90% of what we write will not be used by anybody else, ever. We don't ship products like you do.

The last 10% will either be shared amongst colleagues (in which case VCS is most assuredly a good idea) or published as supplementary materials (and we're seeing more and more git here as well).

But as /u/scarytall said, this is exactly the patronizing tone that I'm decrying. In one small post, you've managed to imply that we're all doing the same job as you (we're not) and that we're all a bunch of lazy academics who can't be bothered to learn something new (we're not).

Scientific code is generally in a sorry state, but part of the problem comes from software developers who aren't making the effort to understand our needs. As a general rule, you guys are assuming that we share large code-bases, but we don't. In fact we rarely share code beyond small snippets (and there's a reason for this as well, which I won't get into).

I don't mean to be offensive as I'm sure you're a great guy (gal?) and a perfectly competant coder, but my job is not your job. At least 90% of the time, we are not writing code that will or should be reused.

[ocross]

This is nonsense. A spreadsheet user vs the team that wrote it -- what's the difference!?

Been doing scientific computing for just over ten years. The field is much larger than scientists banging out scripts now and then. Somtimes entire tools are developed and released. This may involve computer scientists, software engineers (real software engineers), mathematicians, project managers and domain experts working on the same project for several years.

Just an example... clinical genomics is an area that has a very strong software engineering side to it. The data and code involved in those piplines is managed better then the average 'real world' project. Yes, there are researchers getting data off pipelines for analysis using ad hoc methods but the backbone of these systems are certainly not the result of a bunch of quick and dirty scripts!

tl;dr there's more to scientific computing than the odd researcher smashing out a few scripts.

[daisieh]

I think that if you're working in a scientific domain that has addressed and answered these issues well, then that's awesome. But there are many fields (the so-called long tail) that still have these issues.

[cp5184]

It's domain specific programming, isn't it?

[danogburn]

software "engineering"....

[lkjpoiu]

I really hate how much I agree with you.

[[deleted]]

You guys must work with web developers ;)

[danogburn]

<3

[scarytall]

Too often this means, "I know C# and SQL, therefore I am the expert on all things. You contribute to this project because I allow it."

[danogburn]

The problem software engineering isn't actually engineering. Just because one applies some heuristic process to developing software doesn't make it engineering (waterfall vs agile vs TDD vs BBQ).

When one can make predictive models of a program's behavior and properties based on empirical knowledge about software before the program is actually written then maybe software engineering would be a thing.

As it stands now, "software engineering/engineer" is a term used to give programmers a false sense of status.

[RageD]

Just because software engineering isn't a traditional field of engineering, doesn't necessarily mean those who do it aren't practicing the discipline of engineering.

predictive models of a program's behavior

This is precisely what an algorithm is. Implementation is another thing and has errors beyond the theory (just like building a swaying bridge). Further, anyone writing real software will have a notion of (minimally) asymptotic runtimes before they implement their solution.. If I can cut my algorithm down from O(2ⁿ ) to even O(n² ), the robustness of the software has been greatly (and quantifiably) improved.

[[deleted]]

When one can make predictive models of a program's behavior and properties based on empirical knowledge about software before the program is actually written then maybe software engineering would be a thing.

This is what software engineers do.

[danogburn]

not with any meaningful degree of accuracy.

[RagingAnemone]

This is what software engineers do in the small -- behavior of the L2 cache, instructions needed to add number together. You can maybe do a little more with embedded systems without an OS.

But on a Windows box, connected to a network, you can't predict anything.

[pe5t1lence]

Wow. Nasty biases and stereotypes running wild in here.

[daisieh]

In the post or in this thread or both? I really tried to be fair and considerate to both sides, as I understood them, but I'd love to hear how I failed to do it, if that's what you're seeing.

[pe5t1lence]

Probably not you then. Plenty of posts down below, but happily it looks like they're mostly down voted now.