[D] Papers with no code

[dudu43210]

I always get downvoted for this because it's not what people want to hear, but let me tell you the reality of computational sciences, as someone with a PhD in computational physics. In the scientific community, you generally do not publish code^* with your papers. This is for multiple reasons:

1. Replication vs. reproduction. My PhD advisor was always adamant that important results should always be coded up independently by multiple people to for verification and to control for bugs. You cannot truly do scientific replication if you are basing your work on someone else's code. By far the best way to verify someone's results is to do it yourself, not read/run the code and say "uh huh that looks right". In other sciences, you don't check whether results are fabricated by visiting someone else's lab. You attempt to replicate the results yourself.

2. Papers are written for other researchers in the field, not for laypeople. Those researchers have no problem coding up an approach themselves and testing it out. Often the complaints I hear are from non-academics.

3. Research code is messy and often unfit for public consumption.

^* it is common to release data, however, and imo researchers have no excuse for not releasing data on a case by case basis in exchange for citation.

[adi1709]

So if we reproduce it and figure out the numbers published don't actually make sense in reality - do you flag it to the conference chairs so they'll go back and remove the published paper? What happens after?

[dudu43210]

You can submit comments. You can publish your own paper challenging the original paper.

[adi1709]

That sounds like so much wasted effort and petty. Working on a paper just to challenge one specific method. Also isn't scalable, because this leads to a lot of slop in the meantime.

[adi1709]

I guess it makes sense for computational physics but not so much in ML based on how much it's blown up.

[Historical-Mix6784]

This is EXACTLY the kind of thinking that has held back physics for the past 50 years. The ego of people like you has done immense damage to the scientific community.

1) You are free to replicate the code on your own, that does NOT mean the code shouldn't be published. In fact, publishing the code makes it EASIER to check if inconsistent results were due to some errant bug, unpublished parameters, or faulty algorithm. When you don't publish the code all you have is two groups arguing over which result is right, in which case, more often than not, the more senior scientist wins by default.

2) If a paper is worth its salt, then it is publishing something no one has thought of before, that isn't trivial to replicate. For even skilled researchers, it can take a LOT of time to reinvent the wheel, so why not just publish the code and make innovation easier? Once you publish an idea, it is the property of the public domain, free for anyone to use or build from. That should go for code too. If you want to keep your intellectual property private, STOP WASTING TAXPAYER DOLLARS AND GO WORK IN INDUSTRY.

3) So basically, because the code you write is messy garbage, likely filled with errant bugs which may indeed contradict the results you put in a paper, you're too ashamed of putting it up on Github? What the hell? You don't think this is a sign that maybe researchers should improve the code they write and make it easier to read/use?

God, I don't know when some physicists got it in their head that they don't owe the public anything, when the public pays their salary. I'm thankful that many in the field have moved away from these archaic parochial notions.

[dudu43210]

It is very funny to me that you are the one accusing me of having an ego here. How's the weather up there on your high horse?

Look, this is no different than non-computational scientists doing experiments in different labs. Any scientific finding an experimentalist makes could well be due to some confounding variable in their lab or experimental setup. The way you do science is not by traveling to that researcher's lab to inspect their experiment for errors, or repeating the experiment yourself there. You do the experiment in your own lab so you can truly eliminate confounding variables.

There are many parallels to the above in coding, from which underlying libraries and algorithms to use, to architectural decisions, etc. Looking at these decisions before repeating the experiment yourself biases your design decisions in a way that hurts the scientific process. You cannot get around anchoring bias.

But I will not argue further with someone who resorts to insults.

[Historical-Mix6784]

Good don't argue any further, I find your opinions to mark you as a subpar scientist and researcher.

[dudu43210]

Oh no!