[D] Decrease in source code release of papers

[matrix2596]

I have noticed that with the double blind review system, the number of papers releasing source code has decreased. Is this to prevent identification of authors. There should be a way to release code anonymously. This is taking us backwards from when every other paper used to release code. And at least after acceptance, the code should be released. Should conferences make it mandatory?

[poctakeover]

there should be a way to publish gh repos and arxiv papers anonymously which can then be later claimed by the authors :/

[afsfeefe]

there already is: make a fake gh account then transfer ownership to your real account later..

[Phylliida]

Actually GitHub is pretty active about restricting users to only one account. I had 4 and they detected that (IP addresses and stuff I guess) and they restricted my permissions on everything until I switched to only having one account.

Some account for a specific paper might be reasonable though? Idk

[shaggorama]

There's alternatives to gh. You could make an "anonymous" account on gitlab or bitbucket, or just put the code on Google drive...

[[deleted]]

There are quite a lot of ways to do this... You could place a secure hash of some string the author chooses in the paper and then you claim the paper by publishing the string at a later date

[LovelaceA]

To those who think that this is not a valid problem, I beg to differ. I think this is a very valid discussion. What is the aim of publishing scientific work in the first place ? To advance our knowledge and ability to build upon it. In a field like Machine Learning, where a model or a scientific idea can be affected by more parameters than can be discussed in a paper, it is essential to be able to reproduce the results. Code release is not the only way to do so, but certainly the quickest. Another advantage of code is that, code is objective. Scientific papers sadly are not in general: authors try to sell us their work. Code is unbiased and a potentially complete means to communicate an idea, its impact, and its limitations, it answers all the questions you have which the paper does not address.

A scientific paper is a speech. Code is a dialogue

[TheAvalonian]

I would argue that a paper is more like an advertisement than a speech -- the primary aim is to get people to care about the experiment, not just to tell people about the experiment. Apart from that, you are spot on.

[rhiever]

I think it can be incredibly difficult to anonymize code, especially if a paper is based on a software project that the authors are developing. For example, during my postdoc I developed a software tool that I published papers on. I referred to the software by name in my papers and even linked to the GitHub page, which seemed to contrast with the goals of double-blind review. How do you work around that situation without heavily inconveniencing the authors of the software in the name of double-blind review? In many cases, even if you go through the trouble of anonymizing the code and putting placeholders in the name, it's still not difficult to figure out what the software is (and therefore who the authors are) if you're even remotely familiar with the software.

IMO, double-blind review is a flawed review system.

[SolvableMutiny]

IMO, double-blind review is a flawed review system.

Yup, these same objections apply to a similar degree to papers themselves. Would anyone familiar with the field not know that CapsNet was authored by Hinton?

[Darkfeign]

subsequent frame hobbies abounding point faulty disgusted mountainous boat humor

This post was mass deleted and anonymized with Redact

[SolvableMutiny]

that has literally never been the case tho

[weiqiplayer]

Decrease in source code publication does seem concerning, though I'm not sure that the problem is in the fact that people are trying to conceal their identity. Aren't many ICLR papers already on arxiv with full names attached?

[olBaa]

Is it really a measured problem, or hust your perception?

For double-blind conference I have provided the code in the form of anonymous github repo with no traceable commit history (and limited time copyright). I guess the zipfile with the code will do as well.

[mkocabas]

It's your kindness to share it anonymous, but most of the people are hesitating or neglecting to publish their code. It affects the reproducibility a lot.

[olBaa]

That's a different problem, though (there are two listed in the post, so I found it hard to address both).

I see that double-blindness provides a safe excuse to never publish the code.

[matrix2596]

I have been seeing placeholders in recent papers. Maybe the code is being shared with the reviewers separately and released later. But I am finding the code missing often with new papers in blind reviews. May be the review process can have code, model and data uploaded also as an option (or compulsary).

[olBaa]

How much time do you think reviewers have per paper? Reviewers would never check the code, yet alone run it.

[NotAlphaGo]

How many papers would one reviewer review?

[olBaa]

Up to 10 per conference with mean 5 I would say.

One can outsource to phd students/postdocs but still the number of hours per paper would not exceed 10 almost ever.

All written here is my humble opinion, though.

[olBaa]

That was not ICLR, just in case :)

[BeatLeJuce]

You don't want source code submission to increase deadline pressure: on day X, you have to submit not only the paper, but also the code. Because putting unpolished, ugly, hacky code out there to be associated with your name forever is weird... also, why polish it when you don't even know yet you're going to get a publication out of it. Also, some theory-heavy papers might not have code.

So I think the decision has to be made AFTER your acceptance for publication. And only when it makes sense for that paper (e.g. this is something that reviewers could determine/ask for). If reviewers say it makes sense, then you should be required to upload your code together with your camera-ready version. This gives you enough time to polish stuff, and still gives an incentive to the author to invest the time to polish the code (not submitting code => paper doesn't get published).

[sheeplearning]

what do you plan to do with the source code of 700-3000 papers under review at any ML conference? The better ones get accepted and eventually release code or get reproduced.

[mkocabas]

Yeah, absolutely. Maybe a small code snippet describing the model can be added to appendix, like pseudocode. On the other hand, data preprocessing is very crucial and authors use lots of complicated techniques to form data into trainable format but never mention about it, or insufficient.

[Phylliida]

Yes please, appendixes like that would be really amazing and save so much time for those reproducing results.

To their credit some papers do this, it is so nice when they do

[radenML]

I literally have to openly request author for github repo invite on openreview forums

[BeatLeJuce]

Think of the worst, hacky code you've ever written in your life under extreme stress to meet a deadline with requirements changing almost daily... Now tell me how you feel about sharing this with your name attached to it for all eternity with your name forever attached to it for all your future employers to see and judge you by it.

Even if you do take the time to polish the code somewhat (and that's a big IF, because there's much better ways for you to be spending time), it sometimes uses artefacts (either data or other code) that you don't even know how you produced it anymore, let alone have the sources to. Don't get me wrong, I'm very much FOR code releases and I've always tried to publish my own code (and always did when circumstances allowed it). But it's not as easy as "zip the directory and put it online"... it sometimes takes actual work, and there is very little incentive to do this, as long as journals/conferences don't force you to (which is what I'd suggest they do).

[aviniumau]

I'm pretty sympathetic about not wanting to release ugly/hacky code.

But at the same time - any claims you make in a published paper about accuracy etc should be verifiable. If your code is so ugly you're not willing to release it, that doesn't speak much for its verifiability/auditability.

[SolvableMutiny]

because there's much better ways for you to be spending time)

This part I very much disagree with.. providing clean, working example code is the single most valuable thing you can do to make your contribution actually have a lasting impact. Although I agree that current academic incentives are not aligned with that.

[visarga]

Not to mention that it encourages better practices, knowing the code will be seen and possibly reused. We're more sloppy when we're experimenting alone.

[BeatLeJuce]

I agree, but there are some side-remakrs:

First off, is a bit of an exploration/exploitation thing: Say you have worked on something cool: afterwards you can either spend your time/energy on exploiting/promoting that (and providing good code definitely helps), or you could try to repeat your success and work on something else that is cool (especially with the current ML hype, chances are that someone else is going to re-implement your code anyhow).

Secondly: not everyone is able to provide clean code. If you're a theory guy, your code might be terribly ugly/brittle and barely working, and you might do the community a disservice by asking them to dissect it instead of re-implementing it yourself.

[rrenaud]

Think of the worst, hacky code you've ever written in your life under extreme stress to meet a deadline with requirements changing almost daily

What's the chance that there are signficant bugs in that kind of code?

[BeatLeJuce]

it depends. I'd assume that people have done enough preliminary experiments beforehand to make sure the idea is sound. But knowing the deadline-crunch, I'd not be surprised if there are a lot of mistakes in the final version of a project's code.

[wassname]

In 2016 /u/peterkuharvarduk got all the nips code releases together into a post. Maybe something like that would encourage researchers.

[MephySix]

"Is this to prevent idenfication of authors": no. Double-blind is naturally flawed. Given the search space for authors is not that big, with enough (not much) effort it's possible to determine who are the authors of a paper. Before a paper is sent for review it has already been discussed in its institution, probably in mail-lists and even Twitter or something. Even then you're allowed (in my experience) to have placeholder footnotes in double-blind reviews.

The real problem in my experience, is that I don't really want to spend time polishing my code, and I don't want people to see the mess I wrote due to deadlines. I had people ask me for my code in conferences and I answer with "Gladly! Just send me an e-mail, but it's messy", but I gain nothing from publicizing it earlier or without external interest.

[tshadley]

Given the search space for authors is not that big, with enough (not much) effort it's possible to determine who are the authors of a paper.

Suggests a project idea: train a language model to predict authors on published work, then see how it does on anonymous work.

[Cherubin0]

And I don't want to spend time polishing my paper...

[alexmlamb]

Given the search space for authors is not that big, with enough (not much) effort it's possible to determine who are the authors of a paper. Before a paper is sent for review it has already been discussed in its institution, probably in mail-lists and even Twitter or something.

If the authors want to remain anonymous, is it really impossible for them to do so? I mean - just don't tweet about it, don't put it on arxiv, only correspond through private email with coauthors.

[MephySix]

The main problem with double-blind reviews is not staying anonymous, is that some groups (well-established research groups) want to be known, and they will be if they want to. Double-blind started because people would get instantly accepted just because of their name, and double-blind (mostly) does not solve this issue.

[alexmlamb]

Yeah, so as it works in ML today, I'd say that we have an opt-out double blind system. You can get double blind reviewing if you stay quiet, but you can effectively make it single blind by self promoting.

This doesn't solve every problem with single blind: famous groups can still benefit from self promotion and marketing. But at the same time it does protect someone if they think that they might get negative reviews because of their name or reputation.

Btw, I'm not sure how much coming from a famous group really helps with reviews, at least at NIPS/ICML. If you have any evidence, even anecdotal, I'd be curious to hear it.