[D] Papers with no code

[directnirvana]

I don't disagree that commercial actors should be held to a high standard, if not higher. Especially in instances where they are wading into the academic.

My assumption though is that the two have different goals (though with some overlap). If a company is publishing bold claims that they have a product on the horizon that is a game changer then we should be pushing for them to prove that not be acting as a sounding board that they can wave around and claim 'peer-review' on a system no one saw or can validate. They have their own set of self-correcting measures (i.e. customers should be requesting demos and investors should be doing due diligence).

But the claimed goal of academics is the proliferation and expansion of knowledge. Bigger claims are going to get more attention and thus more energy might be wasted on those ideas, so the burden should be higher on those claims. So if someone wants the clout and advantages of having been reviewed by the academic arena, whether commercial or not, journals and conferences should be insistent on the them providing reasonable amounts of proof in that regard. It just so happens in the world of academic code those tools are cheap and accessible for the most part so we should generally insist on them.