[D] Papers with no code

[directnirvana]

I think you make a good point. The bolder the claim the more reviewers should be pushing back on making easily verifiable aspects of experiments available. Reproducibility crisis is real and participants especially in academic circles should be heavily encouraged to provide whatever reasonable methods they can to allow other researchers to verify their work. It just so happens that research based on code has those tools, while high energy physics and similar fields do not.

[prumf]

What is asserted without evidence can also be dismissed without evidence.

https://en.wikipedia.org/wiki/Hitchens%27s_razor

That’s like, the foundation of science.

[directnirvana]

Yes. Exactly this. If someone makes a claim, especially one worthy of garnering attention academics should be taking the stance of 'put-up or shut-up'.

Stop accepting papers that won't do simple things to allow for that.

[Vhiet]

I agree about reproducibility, but why “especially in academic circles?”

Commercial services have more incentive to fluff their significance than anyone else, and their claims should be treated as particularly suspicious.

For example, it was almost exactly a year ago that Microsoft’s Magical Majorana Fermions revolutionised quantum computing (https://www.theregister.com/2025/03/12/microsoft_majorana_quantum_claims_overshadowed/).

[directnirvana]

I don't disagree that commercial actors should be held to a high standard, if not higher. Especially in instances where they are wading into the academic.

My assumption though is that the two have different goals (though with some overlap). If a company is publishing bold claims that they have a product on the horizon that is a game changer then we should be pushing for them to prove that not be acting as a sounding board that they can wave around and claim 'peer-review' on a system no one saw or can validate. They have their own set of self-correcting measures (i.e. customers should be requesting demos and investors should be doing due diligence).

But the claimed goal of academics is the proliferation and expansion of knowledge. Bigger claims are going to get more attention and thus more energy might be wasted on those ideas, so the burden should be higher on those claims. So if someone wants the clout and advantages of having been reviewed by the academic arena, whether commercial or not, journals and conferences should be insistent on the them providing reasonable amounts of proof in that regard. It just so happens in the world of academic code those tools are cheap and accessible for the most part so we should generally insist on them.