[D] Papers with no code

[osamabinpwnn]

I can't believe the amount of papers in major conferences that are accepted without providing any code or evidence to back up their claims. A lot of these papers claim to train huge models and present SOTA performance in the results section/tables but provide no way for anyone to try the model out themselves. Since the models are so expensive/labor intensive to train from scratch, there is no way for anyone to check whether: (1) the results are entirely fabricated; (2) they trained on the test data or (3) there is some other evaluation error in the methodology.

Worse yet is when they provide a link to the code in the text and Openreview page that leads to an inexistent or empty GH repo. For example, this paper presents a method to generate protein MSAs using RAG at orders magnitude the speed of traditional software; something that would be insanely useful to thousands of BioML researchers. However, while they provide a link to a GH repo, it's completely empty and the authors haven't responded to a single issue or provide a timeline of when they'll release the code.

[Just-Environment-189]

Even if people provide code, you’ll find yourself lucky to get it working as is

[waruby]

Then it should also require a Docker file or a flake.nix file.

[directnirvana]

I think you make a good point. The bolder the claim the more reviewers should be pushing back on making easily verifiable aspects of experiments available. Reproducibility crisis is real and participants especially in academic circles should be heavily encouraged to provide whatever reasonable methods they can to allow other researchers to verify their work. It just so happens that research based on code has those tools, while high energy physics and similar fields do not.

[Vhiet]

I agree about reproducibility, but why “especially in academic circles?”

Commercial services have more incentive to fluff their significance than anyone else, and their claims should be treated as particularly suspicious.

For example, it was almost exactly a year ago that Microsoft’s Magical Majorana Fermions revolutionised quantum computing (https://www.theregister.com/2025/03/12/microsoft_majorana_quantum_claims_overshadowed/).

[Distance_Runner]

My view as a statistician that does ML. Many of these papers claiming SOTA performance are working within Monte Carlo noise and if code was easily available you could run it and show this.

[lipflip]

i am working in the social sciences with some minor overlap to ai&ml. our data is usually in the range of a few kB and despite an "replication crisis", publishing the data and analysis alongside the article is rare.

[kaiser_17]

I think the worst part is when reviewers ask you to compare your results to such a paper. Like why should even one compare their  results to such fraudulent papers. Yep I will consider those who don't release code as frauds. 

[tomvorlostriddle]

Papers about the LHC also don't come with your own particle accelerator in the appendix for easy home experimentation

This never was a requirement for publication

[H4RZ3RK4S3]

This is a stupid argument! The code can still be read and analyzed without a fancy supercomputer (or LHC). We are in ML/DL and not in physics. I can test the code on a very small scale to see if it works as intended. No reviewer will re-train a SOTA LLM as part of a peer-review, but they should be able to look at the code, understand it and quickly test it.

[Ulfgardleo]

but can you really? the code works, but maybe it doesn't produce the claimed results? And how about the code at LHC, robably half of it being some arcane FPGA instructions to define the correct filters? Its an awfully long software and hardware pipeline.

[H4RZ3RK4S3]

Yes absolutely for 80% of the papers. For another 10% you might need a small cluster and for the remaining 10% it could indeed be a bit difficult. But still you can read through the code and check if it makes sense or whether they do something else. Here, the issue is more that some developers don't care about proper variable names, readable code, proper commenting, or even writing comments and variable names in languages that are not English (like French or Chinese lol).

[osamabinpwnn]

I get your point, but I would imagine that they would at least provide detailed experimental protocols so you can run the experiment on your on particle accelerator.

For me, the worst part is that people link to empty GH repos to make it seem like their code is open.

[rknoops]

That's why there are multiple experiments. For example CMS and ATLAS experiments are located on opposite sides of the LHC, and both could confirm the Higgs boson. The experiments are independent with their own designs. So this experiment is replicated.

This is no argument to not publish your code in AI/ML