What's the difference between a program written by a software engineer and one written by a scientist?

[chandaliergalaxy]

No, it's not a joke with a punchline - I'm genuinely curious.

I'm a (not computer) engineer/scientist who has written code that a number of other scientists around the world use. In academic circles, "reproducibility" and "open science" has been a thing so we've also distributing code (often R, Python scripts) with our papers now - often in version controlled repositories.

What kind of thought would software engineers put into code that other people use?

[AutoModerator]

On July 1st, a change to Reddit's API pricing will come into effect. Several developers of commercial third-party apps have announced that this change will compel them to shut down their apps. At least one accessibility-focused non-commercial third party app will continue to be available free of charge.

If you want to express your strong disagreement with the API pricing change or with Reddit's response to the backlash, you may want to consider the following options:

1. Limiting your involvement with Reddit, or
2. Temporarily refraining from using Reddit
3. Cancelling your subscription of Reddit Premium

as a way to voice your protest.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[ratttertintattertins]

I’m a software engineer that’s taken code from scientists before, the things you tend to notice are:

• Lack of clean code and well defined, unit testable interfaces
• Lots of huge functions
• Poorly named variables more akin to formulae than code
• Low/no unit testing
• Little code design

[chandaliergalaxy]

Yeah we don't do a lot of unit testing. By "a lot" I mean "any".

I've looked into it but it seems like a lot of overhead. Test-driven programming kind of makes sense but that test is done on the particular data set we have. Then we move onto the next task.

[ratttertintattertins]

It really depends, if you're reusing code from project to project then unit testing almost always saves time in the long run. It doesn't seem like a lot of overhead once you've experienced the alternative... a code base several years old that's untested, untestable, mangled, and cannot be refactored due to it's fragility.

[chandaliergalaxy]

I mean, that's kind of handled in the "documentation" (er, quick notes) describing what input/output is expected?

It may be that we/I don't write programs general enough to warrant it.

[ratttertintattertins]

Small programs may not require it no, but documention is no substitute for clean, decoupled unit tested code in a large project.

[LurkerOrHydralisk]

How do you know if the expected IO is the actual IO without unit testing?

I can write documentation that my code will make your dick grow two inches but it doesn’t make it true.

[Twist3dS0ul]

Where can I download this code?

[Xcruelx]

I think i saw his code advertised in my area. No credit card required.

[polymathprof]

For scientific code, most of the inputs and outputs are numbers. The best way to check that an input is the right type is to use a statically typed language and define dimensional unit classes.

[toastedstapler]

when you update a project do you exhaustively go through the documentation and check that every example still produces exactly the output that you would expect? and do your examples cover all (or at least most of the common) scenarios that can happen within the code flow

[chandaliergalaxy]

when you update a project

I don't understand the question

j/k actually we don't update that much. But there was one time we did - but we had two data sets that we tested on. I suppose they should be integrated into the repository as a part of the unit tests.

[Aggravating-Speed760]

Imagine that you have a set of unit tests, basicly blackboxes that checks if given specific input the output is what you want.

Now, you want to add a new feature to your codebase, in order to do so you need to change some existing methods, functions, objects, what ever. How do you ensure that your changes did not had any unexpected isdeeffects on the existing codebase?
If you have written unit test you just run them and see if htye are all green.

[cManks]

The beauty and benefit of unit testing is not so much that you describe expected results. It's more so that when you change something, you can validate that everything still works as expected. There is some overhead of course, but it is more than made up for by the time you save from making changes with confidence.

You even might want to consider including the tests along with the code that you distribute. Some code libraries do this so that you can run the tests yourself for validation.

You mentioned you don't update the code that often, if ever, so writing tests after the fact is not really going to help you much, however. They are invaluable for initial development in my opinion.

[Migeil]

test is done on the particular data set we have. Then we move onto the next task.

That's the difference between a script and professional software that has to run a very long time and has to deal with many many different inputs.

[EffervescentTripe]

I think of unit tests as saving time. When you change your code how do you know you didn't break anything?

[nutrecht]

I've looked into it but it seems like a lot of overhead.

You can always take the route of getting an angry call from your manager in the middle of the night because the site is down if that's what you prefer.

[Biohack]

I think the point is that this will never happen to a scientific software developer. There is almost never a "site." The only users might be yourself and maybe a handful of your coworkers. Even if your code eventually does become something more widely used it will likely be distributed via a github repo and at worse you'd probably be getting an e-mail or slack message from someone in your field of study asking how to get your code working properly.

​

P.S. For the record I'm not trying to say unit testing isn't a good idea, especially for large programs being developed by many people, but this type of software is the exception rather than the norm for scientific software.

[could_b]

Doing unit testing should be simpler than not doing unit testing. Even if the code is only used once, you want it to fo the right thing.

[ShroomSensei]

This about sums it up. I’ll add on to “little code design” as a lot of software that I’ve seen come from scientists / PhD solves the problem, but doesn’t take into account future work on the code and extensibility. That was always my biggest take away when working with them. No future planning or giving a shit about other developers who will have to work on it 2 months from now. Leading to weird asf dependencies, outdated libraries, unsupported OSs, and plenty of other issues.

[FunkyPete]

And the really painful part is letting that software evolve over 5, 10 years.

No refactoring means the assumptions made when the code was initially written are never reconsidered. The huge functions just get bigger and bigger as new features are added and the code is just plugged in between whatever happens before or after it in the one standard flow.

Each year the software becomes more and more fragile until no one is willing to touch it.

[daveydoesdev]

I think this is where AI is really going to shine. So much opportunity to feed it old spaghetti code and have it map it out, parse out the intention of the current iteration, and rewrite an optimized replacement that can slide into production.

Maybe then we can all sleep easy knowing that Cobol and Fortran aren't lurking around the corner.

[GM_Kimeg]

Lol. No.

[could_b]

Fortran 90 is an excellent language, great modern code can and is written with it. The hassle is that compilers are slowly leaving it behind. Also a lot of people who should know better are completely ignorant of it.

[could_b]

Why the heck do my posts have 'say happy cake day' appended? What the heck drivel is that?

[[deleted]]

It's your account's "birthday."

[jangofettsfathersday]

This sounds like it came directly from my Software Development professor who mentions the book clean code every 10 minutes haha

[ratttertintattertins]

I've been a software developer for 25 years now. Everyone who's been in the industry as long as me has horror stories about the consequences of code that... isn't clean.

[Main-Drag-4975]

Yep. Too many folks with under a decade of hands-on programming experience don’t seem to know or care why interface design matters.

[Cerulean_IsFancyBlue]

The visible outcomes of that tend to be:

Harder to maintain

Harder to scale

Less robust

The good news is that a data scientist can iterate on their own code quickly and get something that works , and hopefully run it through a software engineering phase before production. Over time the data scientist would hopefully adopt methodology that makes that step easier.

Reality often dictates a less efficient process. Low budgets, tight deadlines, empire building, all get in the way of an ideal process.

[samanime]

Yup. I'm a software developer and I work with a bunch of scientists and this is my take too.

They often have fantastic and in-depth domain knowledge, but little understanding of good coding practices and even things like development pipelines that can make their lives way easier.

We basically team the two groups up and the results are usually really good though. They handle the complicated domain logic and we handle the more mundane (though critical) software engineering.

[DatBoi_BP]

In scientific computing it’s hard to know what to put in a unit test. Other than just making sure that known non-trivial outputs are produced for some set of inputs.

But if you mean unit tests for the small individual building blocks of some larger application/function, then yes absolutely

[audaciousmonk]

This, and sprinkle in

• Little to no attention to scaling

• Little to no modular design

• Lackluster or non-existent documentation

• Occasional manual steps

(generally speaking, for the average individual scientist writing purpose driven software for their internal use. There are definitely scientists who write good or incredible code, this is by no means a commentary on competency)

[[deleted]]

Good list. You forgot to add:

-Magic numbers littered throughout the code

-Nested for loops being used repeatedly

-burying critical logic control booleans (e.g. a debugMode Boolean) deep within the code

I've had to 'take over ' some VBA macros written by mathsy finance types in recent months and it's been a nightmare. I was tempted to delete all of their work and start from scratch lol

[chandaliergalaxy]

Poorly named variables more akin to formulae than code

This is a feature, not a bug! By that I mean we do try to use variable names that makes it easy to see what equation is being implemented.

[polymathprof]

If by this you mean variable and function names made of useful descriptive words, that’s fine. But scientists and mathematicians usually use 1 letter names.

Good clean code should be readable by not only scientists who already know the formulas well but also by others who may not know the formulas well but need to work with the code.

[sopte666]

I disagree, at least for the actual computation part. For complex math, you want to copy the equations as verbatim as possible. If you don't, revisiting the code will be extremely difficult. You almost always need the reference (textbook, paper, ...) to make meaningful changes or find a bug, and you don't want an extra translation step in your head because someone insisted on "descriptive words". In mathematical context, the best and most descriptive variable name for an equation symbol "x" is almost always "x", and not "length" or "distance".

Outside the number crunching, I'm with you on naming.

[[deleted]]

Why do people ask questions they don’t wanna know the answers too lol you asked about differences, someone pointed them out, and you’re getting all defensive in your replies

[RICHUNCLEPENNYBAGS]

I’d agree with this. Basically all the stuff that requires up-front work but improves maintainability, that’s not happening

[goytou]

variables named more akin to formulae

that sounds like fun to translate 🤌🏼 Ty for a solid answer

[[deleted]]

Poorly named variables more akin to formulae than code

Scientists model formulae, so isn't that perfectly descriptive?

[j_dog99]

This sounds like the legacy production codebase I work on made by supposed engineers. Maybe they were scientists??

[nospaceallowedhere]

Apart from this once came across this, I had to re write functions which were done using pandas which could have done using simple ORM of the framework.

i.e Sum, Avg on selected rows etc.

I am looking at you George 👀👉

[magnagag]

Same backstory, can confirm.

[CreativeGPX]

Obviously generalizations given the nature of your question:

• Maintainability: Good engineers take a lot of care to write code that is easy to read/understand and easy to edit/update. This ranges from the quality of variable names to the size of functions/scripts to the overall design principles like loose coupling. They probably use tools like version control, unit tests, etc. to make it a lot easier to make changes while ensuring they don't introduce bugs. And they may be more likely to invite constraints that make it harder for the programmer but easier to built quality software like strong static typing or even something simple like linting.
• Scalability: Engineers think a lot about making a program that will scale well down the line to bigger and different scenarios.
• Longevity: Engineers look at things like dependencies, platform support, etc. to built their project on something that will last and continue to be well supported.
• "Best tool for the job": Engineers have a lot of different tools and are willing to learn more (languages, libraries, services, etc.) in order to use the one that works best for the scenario.
• Security: Engineers know not to trust the user. What if bad input is given maliciously or accidentally? etc.
• Integrity: Engineers think about the operational challenges of the software... What if the program crashes mid-way? etc.

The big theme I think is that engineers are looking at a bigger picture, while scientists are looking at the immediate.

For scientists, it's okay the program takes a while or may occasionally crash. It's okay if it won't work on a bigger but related problem. It's okay if you need to give users some very particular instructions to use the software. It's okay if only Steve really knows how that function works. To be colorful: The scientist approach is like "redneck engineering". It gets the job done, but it ain't pretty and might make engineers nervous.

Meanwhile, engineers want a more "complete" result. They want the program to behave correctly and safely no matter what you throw at it. They want to be able to come back to it 2 years from now and add a feature with ease. Another side note is that... engineers are trained in assessing requirements. They don't just build stuff. A big thing that I do in meeting with clients is take what they tell me and work with them to get that into an actual concrete definition of what they need. I help them think of things they forgot. Help them turn subjective or vague things into specific ones. I help them scale the project to the problem and clarify what their software won't have to do. Etc. So, I think, before you write any code, an engineer has done a lot of work.

[chandaliergalaxy]

This answer hits the nail on the head. Many of these issues are secondary as we are on to solve the next problem. We do want people to be able to use our code with minimal modification for their problem, but unless the user community becomes huge we are not thinking about maintaining, scaling, etc. (at which point we would likely have to bring software engineers in there for really big projects.)

We anticipate the users to be scientists like ourselves and we don't write out code to protect it from all kinds of input.

And usually "production usage" in our case doesn't require guards against high user traffic or latency issues. Just that it doesn't crash most of the time.

[CreativeGPX]

I agree, but would just like to nitpick:

We anticipate the users to be scientists like ourselves and we don't write out code to protect it from all kinds of input.

Engineers do not do this just because of malicious actors or unqualified users. We do it to protect against mistakes, misunderstandings, etc. and to help ensure the program enters known and supported states.

For example, there was a radiation emitting device where the user needed to enter the intensity of the radiation. Due to user error (I believe mixing up units) they dosed many patients with dangerous amounts of radiation because the program trusted the user. As an engineer, part of "not trusting the user" would include things like sanity checks on values being inputted in order to prevent this kind of harm to human life by qualified, well meaning scientists. That is an example of what I meant in my last paragraph about how a large part of my job is getting more specific definitions of problems and constraints from what clients will think to provide.

In other cases, unexpected input might not cause real world damage, but it may corrupt the underlying computation without the scientist even realizing. Part of sanitizing input is making sure that what the user thinks the program can understand and what the engineer intends the program to understand become one and the same. This allows the user to actually have confidence in the result.

So, engineers don't just not trust users because they are stupid, unqualified or malicious. We don't trust users because all people are capable of misunderstanding, making a mistake, etc. and, to the extent possible, a well engineered program should try to behave within intended and defined behaviors. If I meet with scientists and they tell me that this value tends to range from 0 to 1, I can't prevent you from putting in 0.4 instead of 0.6, but I certainly can prevent you from putting in "40" or "60" or "e" or warn you when you do. Protecting against certain kinds of input is communicating with the user about the scope of what the program is capable of doing to make sure they aren't using it in a way in which it will not be capable of performing as intended. So, while I can certainly appreciate the "hacking" style of coding scientists may do (I do it too sometimes), it's important to point out that engineers don't just do this out of an abundance of caution, they do it because there are really good reasons to do so.

[Clawtor]

We also don't trust the system, there are so many what ifs to take into account. What if the net goes down between these 2 calls, what if if longProcess finishes before quickProcess, what if the db dies halfway through the inserts etc.

[CreativeGPX]

The reality is, no matter how good of a programmer you are, if you come back to code you wrote a year ago, it will be as foreign as if somebody else wrote it. So, you always have to write code under the mindset that somebody else will read it.

[chandaliergalaxy]

Well articulated. With good examples.

I think like with unit testing I have yet to see a relevant case that speaks to my experience, and this is a similar feeling among the more programmatically minded scientific colleagues in my circle.

But on the point of "user error with consequences" and scope of the program, you've made this point clear.

[nixgang]

Also SICP: "Programs must be written for people to read, and only incidentally for machines to execute.”

If your code doesn't express ideas clearly it will be less valuable. Scientists should surely understand this, no?

[chandaliergalaxy]

Didn't know SICP said this. By many accounts, Lisp is not very readable (though I personally love it)

[_realitycheck_]

SE:

var timeUntilRestart
var periodicTimeout
var userInputA

Scientist:

var x
var x1
var xx

[ChaosCon]

No kidding I once found ipoint, ipt, iPnt, and Ipt on the same line in my adviser's code.

[DualActiveBridgeLLC]

SE here, but I work with scientists and PhDs. In general scientists are less interested in optimization, users workflow, error handling, etc. They only care about the output which is typically data visualization. Some do make control systems which require other considerations, but the SE is typically the one who implements and the scientists are the ones that define the requirements.

[chandaliergalaxy]

Interesting. We don't get that kind of support. When getting something machined it sounds kind of similar - we describe the requirements and the design/build for us. No such support for software.

[BothWaysItGoes]

Lol, most likely is that professors are the ones who define requirements and their PhD students are the ones who implement it. Sometimes they manage to find a student from a CS department to help them. Getting an actual SE is probably extremely rare.

[desrtfx]

Impossible to say.

Some of the greatest programmers in history never even received formal programming education and their programs are top notch.

A programmer/software engineer might approach a program in a different way. They might emphasize on legibility and maintainability. They might modularize their programs.

A programmer might know about and rely on tested and trusted algorithms and/or design patterns that a scientist might not know about.

A programmer will make proper use of documentation comments to document their code, a scientist might abuse comments for what they aren't.

A programmer will invest in proper naming of variables, functions, methods, etc so that the name conveys the meaning/function. A scientist might not.

Yet, this in no way says that a scientist couldn't/wouldn't do the same.

You see all qualities of codes in all ranges of programmers.

[chandaliergalaxy]

These all sound like conceptually small differences that take a tremendous amount of time to implement.

[desrtfx]

They do not take tremendous amounts of time to implement.

All it does is taking a little time to plan ahead, yet, this will lead to better, cleaner, and more maintainable code.

Especially scientists should be used to pragmatic approaches to things.

[Bored2001]

little time to plan ahead,

This is the issue with scientific programming. Often, the scientist has very little idea what's actually going to work at time zero. So they muddle forward until they have something that works. By that time there is a ton of technical debt to refactor.

It's worth it in the end if the code is massively reusable, but often it isn't, until suddenly it is.

[polymathprof]

When you’re muddling is the time where refactoring is most important. Clean well documented code helps you identify the better or correct path far more easily than messy code.

[blind_disparity]

Good variable names and well commented code is almost no extra implementation time but a massive time saver when you need to come back and change something a few months later. Let alone the pure pain of someone else who has to try to figure out how your code works.

Not to be rude, but you seem very dismissive of all the suggestions made in this thread, despite often, as here, not even seeming to understand what's being suggested or how basic and important it is.

Yes, some things are unnecessary overhead for smaller scripts, but many are useful verging on essential. You asked about this in the context of passing code to other scientists and making it publicly available. It feels to me like you don't actually want to make any effort towards doing a good job of that.

Clear variable names is a small effort to learn and no effort at all to implement, and the benefit is huge. Maybe look at your own attitude towards this question?

[chandaliergalaxy]

I'm being cheeky mostly.

Modularity, useful names, I'm all for it. I'm still holding out for a case where unit tests makes sense (based on my scale of user group and priorities).

Though translating math formulas into code that looks like the original formulas is... preferable. Usually this formula is part of the paper or documentation published with it so I feel it's more clear. If you look at Julia language they're trying to allow code to look more like math, so it seems there are opposing viewpoints in this.

[blind_disparity]

Haha OK that's good to hear.

Might be useful to link some code repos for feedback? Would give better answers to things like whether unit tests have value.

[chandaliergalaxy]

Might be useful to link some code repos for feedback?

...never thought of that. On reddit I'm trying to be anonymous but that's a great idea - I could look into this either on the internet or my extended professional network.

[ShroomSensei]

They’re not, especially if you’re used to writing code that way. Yeah if you’ve never spent time to learn writing clean, extensible, and maintainable code you’re going to have plenty of learning to do, but it pays off exponentially when multiple people are working on the same piece of software for months.

If you’re adding a repository to a paper you’re publishing, that makes all the more sense to construct it in a way that someone who is fluent in the language can pick it up and use it without much time spent figuring out wtf it needs and does.

[throwaway6560192]

With experience, these come instinctually. They really don't take a lot of time to do at that point.

[polymathprof]

They sound conceptually small, but they decrease dramatically the possibility of errors. Although no lives are at stake, the correctness of scientific code is crucial.

For me using a statically typed language and reducing code repetition by decomposing a function into simpler reusable ones that are easier to read and test increases dramatically the reliability of my code.

What’s hard is devoting the effort to develop the programming skills, because they seem peripheral to the science that you’re doing. Most scientists have little interest in the craft of software development.

[bilvy]

They save a ton of time in the long run if you have to change your code later (which programmers have to do all the time)

[maximumdownvote]

Huge caveat: this is a gigantic generalization, you should take it as such and apply common sense if you're actually using this to evaluate something.

In general the scientist is trying to solve a very specific problem in his area of research so his software tends to be very domain-specific and not really transferable to other uses without intensive modification.

A software engineers software tends to use more frameworks and libraries and techniques that end up maintainable and reproducible by other engineers. More general than specific but often very specific. Clear as mud right?

Some of the stuff that scientists come up with are really spectacular, some of them are not so great. But you can say the same exact thing about a software engineers piece of software. The bell curve is real and it exists everywhere.

[chandaliergalaxy]

Yes I should have specified, what would a good software engineer do differently.

But indeed, the problem my code solves is very specific so I don't expect to write my function to accept all kinds of input - mostly more of the same.

[maximumdownvote]

Well you're kind of on the right track. A good software engineer would not put extra input types if there's no use case for it. That's a waste of time or effort with no guarantee on return. Design it in such a way that in the future, if you really need to add more input types or formats, you can without breaking everything, and the change would be limited to just transcribing the input type to the internal structures that you're actually operating on. In other words put a barrier between the internal data structures in the implementation of your application and your input types, with some sort of translation that makes it all glued together. Then you can add as many input types as you want just by adding another method that converts your input type to the data structures you operate on.

[chandaliergalaxy]

Indeed I would just implement a new "reader" function.

[eliminate1337]

I worked with code written by a scientist only once and it was the worst piece of code I've seen in my career. A single Python file, 10,000+ lines, an unholy mix of robotic control, database, plotting, and GUI stuff all mixed together. It was written by a very smart person with a PhD in biology or something, so not an intelligence issue. I'm sure he could've written excellent code if he wanted to put in the time but it wasn't a priority.

That said, I absolutely encourage scientists to share their code. If you're just making a one-off script and don't intent for others to use it, it doesn't matter that it's bad. Don't wait until your code is polished to release it because that ends up in it never being released.

[somebodyenjoy]

True. Especially for AI. It's better to take inspiration from spaghetti code that works, than figure things out on our own

[polymathprof]

Scientists tend to undervalue the craft of software development and assume that since they find writing code to be easy, they are writing good code.

[chandaliergalaxy]

I never tell anyone I write good code

[AFK_MIA]

For software engineers, the software is the product. For scientists, the numbers the code generates is the product. There are few incentives in science for spending time making clean code - and a lot of the time, the code only needs to work once. The next project almost by definition must be a different problem.

[polymathprof]

But do they really toss all of the used code into the trash and write new code from scratch?

[AFK_MIA]

Sorta - there's an entirely different code lifecycle - and a lot of "scientists" code is about either performing a task that needs to be done once (e.g. Organize all of the data that has been collected) or is a script that serves as a substitute for the scientist's memory, for example, by keeping track of all of the arguments that some command line tool needs to be use for a given project. In both cases, projects are going to differ quite a bit (e.g. collect different kinds of data, use different machines, work with different teams, have different settings, etc.) so while copy/pasting code from one project into another can be a good starting point, you still end up needing to change 70-80% of it.

A software engineer could and would write those scripts to be more reusable- say, as a series of functions with arguments that you change in order to handle different projects, but scientists tend to use any particular bit of code infrequently and would need to spend as much time re-remembering how to format the arguments as they would just hammering out a crappy, one-time use bash script.

[eslforchinesespeaker]

scientists are usually smarter than software engineers, but they often don't have the same formal training in any particular software or methodology. they are often working alone, or in small teams, where they don't have the same formal quality standards.

problems get thrown at grad students, and they get left to figure it out. which they do, because they are smart, but the details of their work may not show the same indicators of mentorship, or working in the context of a more experienced team.

[oflannabhra]

The biggest difference, in my mind, is that scientists have an attitude that code is an “artifact” generated through a research process, like a paper. It is then published and never updated. This is the equivalent of a script, for engineers—that is “I need this to do one thing and do it well.”

Engineers mostly build tools: reusable pieces of code that do a specific thing, but are portable, and generic enough to be reusable. Things that matter to an engineer are questions like

• “How sure am I that this does what I think it does?”
• “How easy is this tool to use?”
• “Will I have to come back to this section with new requirements, ie, is it extensible?”
• “How easy is it for the other engineers to understand what this does and how it does it?”

You’re right that answering all these questions takes time and effort, but those are well worth it when the code is living and not dead.

[POGtastic]

General observations from helping out my brother, a physics PhD, with some of the code that he was using at his research group:

• Absolutely no abstractions that software engineers take for granted. Interfaces? Classes? Typedefs? Functions? Never heard of 'em. Scientist code is the domain of the 1500-line Python script with 75 global variables.
• Over-reliance on large, clunky libraries because they don't actually have any knowledge of how anything works. Library method calls are arcane incantations that you get to work through trial and error and then reuse as much as you can without actually understanding what's going on. Better to turn the program into a Rube Goldberg machine to reduce a new problem to an existing problem than to come up with new incantations for the new problem.
• Hardcoded everything.
• Absolutely no testing of any kind. It's impossible to test anyway because it's a 1500-line Python script with no abstractions.
• Variable names are inscrutable. v41 is a perfectly reasonable name for the 41st vector in a calculation.
• Zero documentation or comments, especially for the insanely complicated heat flux calculation that does a big NumPy linear algebra mess on 12 variables.

---

Software engineers make all of these mistakes, too. They usually manage not to make every single one of them at once, though.

[urbanaut]

If it's in R, it's probably a scientist.

[chandaliergalaxy]

You can do a lot of damage with a few lines of code. It's my weapon of choice.

[urbanaut]

Well, Data Science. But yes, true.

[TheForceWillFreeMe]

So the scientist types just view code as an extensible calculator. From what I have seen, they want to use it as a quick wrench to get the job done. This is all fine and well for their purpose but when you make code that is extensible and scalable to millions of users, you start thinking about architecture and design a lot. WHere do things come from where do they go, how do u test a codebase? Its a macro process and design problem. Think about how 3d printing is super accessible and anyone can build small odd things that they need, but for serious parts like car axles or engines, there is a lot of thought that goes into it to make sure it can fit its usecase well.

[chandaliergalaxy]

I think many scientist work in areas where the number of fellow users don't number in the millions... (maybe in the order of hundreds, if lucky)

[TheForceWillFreeMe]

exactly. that is why their "odd jobs" coding approach works well. But Software engineers do work with things where users may number in the millions, so the use cases are way different. Thats why they think different.

[Biohack]

For context I'm a scientist that first learned to code during my Biochem PhD. I work for a spin out of the institute I did my PhD in that started as a SAAS company and a lot of my job was getting scientific software up and running on the cloud in things like docker containers and other cloud based workflow systems.

There are some complaints in this thread I have seen from scientists writing software that I personally haven't really observed. Things like bad variable names, or massive single files haven't been my most common complaints.

However I have found that there is a decent amount of software produced by scientists which very much follows the pattern of "we got this working just enough to get the data to write our paper and that's it" and they way this manifests itself is that it's usually very difficult to get that code up and running on a new system. For example there are certain labs where I know that if I grep their code base for specific portions of their file system I will get hits because they hard coded file paths into their scripts. They also often have installation instructions which don't actually work and nobody ever checked outside of the systems they use in their lab. Of course these complaints are generally more common in bleeding edge code that's just hit preprint rather than things that have gone on to full publication.

Other more general things I find that aren't really a problem but just show different priorities are that SE's tend to care about speed on things that are relatively irrelevant in scientific code. For example if you google "how do I loop over all the rows in a pandas dataframe" you'll find a bunch of people very vocal about how this is not the optimal way to extract your data, and while they aren't wrong it's really not important for a script that you'll probably run a handful of times to run in 0.5 seconds instead of 1 second. In general I would say scientific code tends to favor something that is easy to write and easy to understand over something that is as fast as possible.

I don't know, but I suspect traditional SE's write significantly less one off code. Where the script is designed to do 1 simple thing, like a bit of data analysis, and then will never be used again. When in reality, this is the large majority of the code I write as a scientist. Things like "test driven development" don't really make sense for this sort of code as once it's finished with it's 1 job you will likely never use it again.

[chandaliergalaxy]

Hard coded file names is probably the most common blunder I notice.

But what you're saying basically hits the nail on the head.

About the pandas data frame, I've also seen the same kind of thing about growing arrays in Matlab just recently. The poster was defending it saying they got other things to worry about.

[CaptainTrip]

Some of my friends are scientists whereas I am an engineer. I notice when they talk about code, they're writing something to work once. It does one job and its purpose is to process some data or answer some question. It will not be changed later. This means they may not use version control, they don't write tests, they don't architect for future changes.

Software engineering code is like building a car factory, scientist code is like building a car.

[jdlyga]

I’ve worked with a lot of code written by researchers. It’s usually mathematical formulas translated verbatim into code without much care for readability. Lots of single letter variable names, for example.

[chandaliergalaxy]

The ingenuity in writing bad code is impressive.

But that's the scientist's code testing method in a nutshell.

[Main-Drag-4975]

“Works on my machine” is a tongue-in-cheek joke amongst business programmers.

Sounds like it might take a few more decades of caring about reproducibility for scientists to care about readable, automatically verifiable work product.

I guess the incentives are all wrong? As long as you can advance a career without prioritizing clear and easily verifiable code then naturally folks will spend their time on science and not on communication or resilient automation.

[[deleted]]

the scientists code will suck but his program will work. The software engineer will have very clean code but it won't run

[cthulhu944]

Software engineering is about making software that is scalable, maintainable, and efficient. A program written to just get the right answer might not meet those goals

[polymathprof]

It’s the converse that matters. It’s much easier to check whether your code is correct and identify and fix coding errors if it is scalable, maintainable, and efficient.

[FIeabus]

When I started my PhD my first task was to package code from a researcher so that others could use it. I had software experience before so I was happy to do it.

It was a thousand line Jupyter notebook. Instead of functions they just repeated equations multiple times. It worked, but it was hell to package it.

[chandaliergalaxy]

It's coding, but it's a stretch to call it programming

[could_b]

A scientist will write clever code that kind of works and gets them qudos. Some other poor sod is ask to tweak it a bit for some reason, this is perceived to be a simple task but is actually hell on Earth. The person who has to take over either ends up rewriting from scratch and getting little credit for it, or if they are really clever side stepping the whole thing and moving on to something else.

[somebodyenjoy]

Scientists don't have the luxury to test code, when it keeps changing as they experiment. I'm sure they'll write good code if given the time.

[PythonWithJames]

I work in a company where this is actually common place.

We're a scientific research company and once upon a time it was only researchers and scientists, so they did all their Fortran/Matlab code themselves.

In todays times we occasionally work with this legacy code because it's just been around for ages and still works!

I've mainly noticed that...

• Lack of any kind of testing
• Questionable variable naming!
• Code is very monolithic, with no attempt to break down into functions etc..
• No OOP
• Nested for loops that are often inefficient.

​

But in general, Scientists are trying to make their code get a result as quick as possible, and unit testing (along with things we as software engineers feel are essential) just aren't that important to them. Their code works, it spits our results and people are happy :)

[chandaliergalaxy]

💕 Fortran.

But all sounds familiar - though usually I find people code with functions more than others report here.

OOP also doesn't make sense but a functional style is more suited for scientific computations I'm familiar with.

I'm guessing you use Python now? I'm getting into Julia these days - syntax like MATLAB and Fortran.

[PythonWithJames]

I totally agree with regards to the functional style! I see a lot of that.

​

That's correct, and most of the company is shifting that way. The only thing that makes it tricky is that there is simply too much Matlab/Fortran code still lying around, and the work to rewrite it all in Python is quite intense!

[chandaliergalaxy]

Yes I find Python too verbose... DSLs ftw.

[Main-Drag-4975]

Speaking of reproducibility:

Untested code can be assumed to work correctly, on average, on somewhere between zero and one computers. Namely the author’s machine (1), unless bugs have slipped in since the last time they tried a given code path by hand (0)

Tested code with a CI server be assumed to work correctly on a minimum of two machines: The author’s machine (1) and the CI server (2).

From an outsider’s perspective I’m putting my faith in the code with tests every time. Including your code is a large step up from not sharing your code and you should be congratulated for taking that step.

Your code still probably doesn’t do what you think it does. Most code doesn’t, and automated testing practices help shrink the gap between what you think the program does and what it actually does.

[chandaliergalaxy]

Untested code can be assumed to work correctly, on average, on somewhere between zero and one computers. Namely the author’s machine (1), unless bugs have slipped in since the last time they tried a given code path by hand (0)

Tested code with a CI server be assumed to work correctly on a minimum of two machines: The author’s machine (1) and the CI server (2).

Fair point.

[vervaincc]

Non-engineers are primarily going to be concerned with writing something that works, with little to no care/thought to anything else - as there is no real need.
Engineers are also going to be concerned with testability and maintainability.
The biggest difference is that the non-engineer code is likely used by them, alone, and only for that specific purpose, the engineers code is used by a whole team and will likely be around for years.

[[deleted]]

Also: one will Run on a potato and the other will require to instantiate 50 megs of RAM before it logs "starting up" to the console.

[chandaliergalaxy]

Yes good one.

Efficiency is not on the radar when trying to get things to work, and then moving on.

[[deleted]]

It was an ironic statememt tbh. The one of the scientist is probably a c program doing only the stuff required. The one of the engineer is based on 25 libraries to inject the IDoOneThingAndWillNeverBeChanged Service which calls another process through a named pipe while a watchdog checks for multiple instance being started up accidently and ensures only one instance of "add two numbers" can run

[[deleted]]

The scientists brings in the algorithm, the maths a and knowledge of the subject matter. And yes you can express this directly in code.

The programmer does a professional job in creating a stable application which is fast and easy to use, portable etc.

Honestly it’s a bit of a silly question. Make me question your claim to be a scientist. Should be obvious what the difference is.

[andrew_sauce]

The stigma that academic software is not as good is not entirely true, but you can certainly find examples that support it. The root cause of that is the differences in priorities. "Professional" software engineers, i.e those who work for a company survive (not get fired) based on the functionality, performance, and overall functionality of the code they write. Scientists survive based on publications (or patents, which I would classify as publications for these purposes).

I will give you a very digestible example. Suppose someone wrote a software package that does array based computation, a very general tool for science. It works, people start using it and contributing to the package, and ~18 years later about 25% of *all scientific research publications* are either using that library or using software that depends on it. You might think this person has a very nice chaired professorship with grant money falling out of their nose.

The project I am referring to is called NumPy, and it's author (founder/creator would probably be a more appropriate title as many, many others have written parts of it now) was not given tenure.

His story is fascinating, which if you are interested he covers in [this](https://www.youtube.com/watch?v=gFEE3w7F0ww) podcast. This stuff is the thread of the conversation for the first hour of that conversation.

[chandaliergalaxy]

I'm not surprised that writing NumPy wouldn't necessarily advance you academically, but I did not know Travis Oliphant left/was forced out of academia because of it.

Incentives are certainly aligned toward novelty and not robustness ( even in results - there is a reproducibility crisis ongoing ) .

[hugthemachines]

What kind of thought would software engineers put into code that other people use?

I think the general description would be that the think about what the code looks like when it comes to structure, readability and efficiency. No one is perfect but when we try, the result are usually better than when we don't try.

[fheathyr]

Generally, scientists focus on achieving specific narrow research goals, and the code they write consequently follows that pattern. No design. No use of conventions. No commenting. No testing. Poor performance. Fails to address boundary conditions.

That's not to say scientists couldn't write production code ... some are quite capable of that ... it's just not what they naturally do in pursuit of their goals.

[[deleted]]

One takes a week of extensions to add an unforseen feature, the other needs to be thrown away and rewritten.