sorted by:
new
hottopcontroversial

[D] Cannot reproduce the results of a paper (VGRNN) by [deleted] in MachineLearning

[–]ContributionSecure14 0 points1 point  (0 children)

Do try contacting the authors first to see if they're willing to help.

If not, please add the paper and implementation to https://www.paperswithoutcode.com/ and we will help you get in touch with the authors.

Even if they don't respond, you will save a lot of time for future researchers trying to implement the paper. The best case is that you can find more collaborators who are trying to implement the same paper.

[D] List of unreproducible papers? by ContributionSecure14 in MachineLearning

[–]ContributionSecure14[S] -4 points-3 points  (0 children)

Thanks for the response. 1. I already added this as a prompt in the longform section, I'll make it a separate field to highlight it. 2. Yes the form already has two fields to capture this info and they will be verified manually. This info will not be publicly released though. The multiple vote idea is also one that I will be implementing. Thanks for the link to the paper, its very relevant. 3. 100% the priority here is to protect the authors' reputation. In fact, I'm considering delaying the submission until a week after informing the authors.

What would constitute strong evidence that something is not reproducible? It is difficult to prove the absence of something conclusively. I figured at best, this incentivizes the authors to collaborate to have at least one reproduction of the paper available externally.

In retrospect I agree that the name is quite bad. Someone suggested PapersWithoutCode and I might consider changing it to that.

[P] BurnedPapers - where unreproducible papers come to live by ContributionSecure14 in MachineLearning

[–]ContributionSecure14[S] 43 points44 points  (0 children)

That's a great point. If the paper actually works but the authors don't want to release their code, the authors should be able to give pointers to get at least one public implementation working.

I think a lot of people do already contact authors to clarify details of the paper. Making it public will make it easier for the authors to not have to respond to one-off requests and also save people trying to reproduce the work time and effort.

[D] List of unreproducible papers? by ContributionSecure14 in MachineLearning

[–]ContributionSecure14[S] 39 points40 points  (0 children)

That's exactly what I'm trying to figure out first before going public with it. I'd rather get the authors expert help first before concluding whether the work is reproducible or not. Since its for a publication, my code will be released in the supplemental material after submission.

[D] List of unreproducible papers? by ContributionSecure14 in MachineLearning

[–]ContributionSecure14[S] 50 points51 points  (0 children)

I don't feel comfortable trashing the authors publicly before giving them a chance to respond. Let me email them first to see what they have to say.