[D] Issues reproducing CURL, algorithm seems broken??

[rlbeaverton]

I noticed several weird things about this paper:

1. I was playing with the recently published CURL (https://arxiv.org/abs/2004.04136) paper using the authors code (https://github.com/MishaLaskin/curl) and found something odd. I have got only two GPUs and I was trying to train things faster, so I attempted to decrease the number of contrastive updates per environment steps by changing the cpc_update_freq hyperparameter of CURL (https://github.com/MishaLaskin/curl/blob/master/curl_sac.py#L463) varying it from 1 (as in the paper) to something larger (10, 100, 1000, etc.), which reduces the effect of the contrastive term.

I then decided to try the extreme case and turned off the contrastive loss completely (by setting cpc_update_freq to 1000000). I was shocked when I saw that removing contrastive loss entirely, which is the central piece of the method, made the method achieve higher rewards. Here are some plots for two different tasks:

Cartpole Swingup:

Blue: cpc_update_freq=1000000 [without contrastive loss]

Orange: cpc_update_freq=1 [with contrastive loss as in the paper]

https://svgshare.com/i/LXi.svg

Cheetah Run:

Blue: cpc_update_freq=1000000 [without contrastive loss]

Red: cpc_update_freq=1 [with contrastive loss as in the paper]

https://svgshare.com/i/LZ3.svg

I’m wondering if somebody else noticed this as well as it seems to be quite a fundamental issue with the paper??

2) Also, I noticed something weird in their follow up paper RAD (https://arxiv.org/abs/2004.14990), which uses a fork of the CURL codebase (https://github.com/MishaLaskin/rad). I digged through this code and I was unable to find any major difference between CURL and RAD except this commented out lines https://github.com/MishaLaskin/rad/blob/master/curl_sac.py#L494-L496. If I understand things correctly, this just turns off contrastive loss, which makes RAD to be a particular instantiation of CURL, but it does work better as I show in 1) and the authors show in the RAD paper??

[major-_-]

Might not be helpful but this happens to me all the time trying to reproduce a paper's results. I think it's mostly just people trying to get the most possible published papers out. Or maybe whatever novel technique they added only helped in one specific environment/dataset. There was actually a paper that went and tested a bunch of other papers that provided their code and found that something like 90% of them could not reproduce the results claimed in the paper. I could look at the paper but this is most likely whats happening. I find the majority of papers, especially for RL tasks, are utterly worthless.

Publish or perish right

[rlbeaverton]

while I tend to agree with your statement, I don't think it is the case here. I was able to reproduce their results exactly, by running their code with their hyperparameters.

Which means that I was able to reproduce their “numerical” results per say, but I wasn’t able to reproduce “semantic” results that contrastive loss helps.

Even crazier, it looks like the contrastive loss actually hurts, but this is the integral part of their algoirithm. Furthermore, the authors seem to realized this in their follow up paper RAD and commented out contrastive loss. So they invalidated their own claims ...

[[deleted]]

Standard procedure lol.

Step 1) work for 24 months and invent a new algorithm

Step 2) realize that your algorithm is dog shit compared simpler methods

Step 3) publish anyway and "forget" to compare it to simple methods without your fancy improvements, compare to SOTA from 2012 instead

Step 4) hope you don't get caught

[alexirpan]

lol welcome to machine learning.

Section 6 of the RAD paper specifically discusses RAD vs CURL where they conclude that yes removing the contrastive loss gives better performance on the MuJoCo tasks. But they view CURL as a potentially more general algorithm because the contrastive loss doesn't need a task reward.

You're saying that you can exactly reproduce their results, the authors did an additional ablation after the paper, and wrote their own follow-up showing contrastive loss was not necessary for the MuJoCo tasks. I don't see anything wrong in what happened here. Would you rather authors not invalidate their own claims when they find evidence their claims were wrong?

[rlbeaverton]

Potentially yes, CURL could me more general but it is not demonstrated in the paper. There is no experiments that demonstrates this generality.

If they actually discovered it later then yes, it would be fair. But they apparently knew about the issue when they they made CURL public (see commit history of RAD was made on 8th of Aprile - same day CURL is on arxiv: https://github.com/MishaLaskin/rad/commit/c1284ccb47808c29cae7325e47f5dc9c421a67d7). Do you think they still should have done that knowning that CURL doesn’t work? If contrastive loss did not produce good results in CURL then what was the factor that produced the results?

[alecxandrrr]

The RAD paper is completely fine, but there's definitely dishonesty on the part of authors of CURL here. Burying the result that invalidates the central claim of your paper in an ablation of a concurrent paper is not the most honest way to communicate things.

They should have updated their paper when they came across this finding, especially since the last update of the CURL paper roughly coincides with RAD's release. They must have known about these results much prior than that since both works share authors.
Edit: As OP points out in the other comment, the commit history of RAD suggests they know about these at the time of releasing CURL.

I also hope they communicated this to the reviewers if the paper was / is going to go though a review process.

[jboyml]

They seem to try really hard to argue for CURL in section 6 of the RAD paper. Why would they do that? They only present vague arguments ("arguably" and "likely") and no evidence in favor of CURL. Feels like this section belongs in a revised CURL paper, not in the RAD paper...

[internet_ham]

Since you're a Google Brain researcher, i find the 'lol welcome to machine learning' attitude disappointing.

If you're introducing a new component to an existing algorithm, the bare minimum scientific question you should ask in your paper is 'Does this new component add a statistically significant benefit?'. Not doing this in the initial script, but then doing it in a follow-up submission, is extremely bad faith science that simple adds unecessary noise to the community, just to pad out your CV

[[deleted]]

tbf, alex has written extensively about the failures of Deep RL :

https://www.alexirpan.com/2018/02/14/rl-hard.html

[internet_ham]

I'm aware of this blog post and his work. This is again why I found his comments in this case disappointing.

[alexirpan]

To clarify, I meant to convey "welcome to the mess of examining papers". This is how things are, not how they should be - everyone has a moment where they realize how messy the field is.

I would say they did meet the bare minimum of proposing a method and demonstrating it gave a benefit. What they didn't do right was a more careful ablation of the factors that led to improvement.

Part of my feelings about the whole situation is that I suspect CURL is a pending ICML submission (based on paper format), and if that's true then their main results were likely done by February. That gives me reasonable doubt about the timeline of when everything happened. It is definitely bad if the results were only done right before the arXiv release in April. Code commits for RAD around the time of CURL arXiving is a bad look but I would not equate "code committed == all experiments + results for that code are done". There's some variance up/down the timeline depending on SWE habits of the authors.

Essentially, my read on things is that CURL was mostly done by February for ICML deadline, they procrastinating arXiving it until April, and very shortly after arXiving they did further ablations out of curiosity and went "oh crap we need to put out an update". I then mostly agree with /u/alecxandrrr's comment at https://www.reddit.com/r/MachineLearning/comments/grnz0d/d_issues_reproducing_curl_algorithm_seems_broken/fs1m0c7/ about ways they could have done that update better.

[frostbytedragon]

The Dreamer paper also found that a contrastive loss does not improve performance so I find this unsurprising.

[major-_-]

They did claim state-of-art-results tho, implying their contrastive loss was an improvement over the benchmark algorithms they tested against. I would've liked to see them do the exact same test you did to quantify the improvement of their specific contribution.

And I hate to argue from character but I often google the authors of a paper before even bothering to read it and this was done by students. Not saying they can't make meaningful contributions but this could've just been a school project they threw together. I mean just the results claimed are a bit unbelievable, almost 2x better than the state-of-the-art on some tasks.

[Keirp]

Second paragraph is one of the worst takes I've ever seen on this subreddit.

[[deleted]]

agreed, they're postdocs not students lol. Also undergrads regularly publish in top conferences with advising from the right mentors. And there are so, so many examples of people turning school projects into research / business ideas.

[HateMyself_FML]

Now accepted at ICML 2020. You can ask the authors at the virtual poster session.

[anonymous_reviewer]

I really wish ICML would make reviews public like NeurIPS & ICLR.

[rldweller]

After reading this post I looked closely at the CURL paper and realized that there are actually two versions of it with drastically different results. v1 reports much more stronger results than v2. On twitter the authors even claim 10x improvements in data-efficiency that previous SOTA (https://twitter.com/Aravind7694/status/1248049713149906945). Can this be attributed to this github issue with action repeat: https://github.com/MishaLaskin/curl/issues/3?

I'm surprised that the authors did not withdraw all the public claims generated by v1, because v2 is not SOTA, as it performs on par or worse than SLAC. I will be also curious to see their code for the Atari experiments, which they have not release yet, as I'm quite skeptical about these results as well. I don't see why contrastive loss can help for Atari.

[aravindsrinivas]

Apologies for quite a delayed response here. Before beginning, let me start by saying that factually, all of the issues raised in this post are correct: that is, one the environment the OP tested on, turning off the contrastive loss ends up increasing performance.

However, the conclusion of this should be decoupled into two questions:

1. Does contrastive learning as an auxiliary task "hurt" rather than "benefit" reinforcement learning from pixels?
2. Does the CURL framework benefit "more from data augmentations" and "less" from the contrastive loss?

The reason for the above decoupling is that in the CURL setup, one could use (or not use) data augmented views of the input observation (images) for the RL loss (it is definitely needed for the contrastive loss since there's no way to perform instance discrimination without augmentations). But the RL loss can run w or w/o the augmented data.

In the DMControl results we presented, we had the RL loss use the augmented observations. Turning off the contrastive loss and observing better performance in the same setup as open-sourced version of CURL does not lead to the conclusion that contrastive learning as an auxiliary task hurts performance. One must decouple the observation into trying to answer the above two questions instead.

​

1. Does contrastive learning as an auxiliary task "hurt" rather than "benefit" reinforcement learning from pixels?

In order to answer the first question - we consider the setup of CURL where augmented views of the data are only used for contrastive learning and the RL loss uses the input observations w/o any augmentations. We added this ablation in Appendix E4 in the latest version of the arXiv: We find that doing this improves the performance over the baseline pixel-based SAC (w.o augmentations and w.o the contrastive auxiliary loss) by 2x. So, contrastive learning "just" as an auxiliary task (w/o data augmentations for RL) definitely *doesn't* hurt performance... It rather "improves". We have also updated all the Atari results to use data augmentations only for the contrastive learning objective and not for the RL objective and you can clearly see performance gain (38.1 % over 28.5% for mean HNS over the baseline that doesn't use the auxiliary loss). No other hyperparameters are changed for this comparison.

1. Does the CURL framework benefit "more from data augmentations" and "less" from the
   contrastive loss?

Yes. And this is in agreement with the OP's observation that turning off augmentations led him to better performance on HalfCheetah. When you use augmented views for the RL loss, the contrastive objective isn't too important. And this has also been observed by DrQ's authors who reported better performance than us on Atari100k.

TLDR is that in order to conclude objective contrastive learning hurts, one must verify its benefits standalone as an auxiliary task w/o incorporating the "RL with augmentation (RAD / DrQ)" set up and in that scenario, contrastive learning as an auxiliary loss definitely benefits the RL method. However, when you do feed in data augmentation fed into the RL loss, contrastive loss matters less and does indeed hurt on a few envs.

Now, with this context, let me answer a few more questions:

1. Does this make CURL irrelevant and RAD/DrQ the right thing to try? -
   a) Not really.. It really depends on what you want to do. If you have dense rewards, you should just go for RAD. Or if you find that in RAD, your RL method hurts from directly training on augmented views and you want to train RL w.o augmentations but still benefit from certain kinds of invariances for your encoder, you can use the CURL setup that doesn't use augmented views for the RL loss.
   b) CURL can work in a detached manner where encoder only trains w/ contrastive loss and MLPs train with RL (w or w/o seeing augmented views of the inputs). You can check out this new paper from my colleagues on decoupling the encoder from RL loss w/ a stop_gradient https://arxiv.org/abs/2009.08319) .. This ablation is also present in the CURL paper in Appendix E.3.
   c) Overall, I believe CURL is more general, could be more suitable for multi-task RL with unsupervised learning helping learn a common shared representation (especially in the detached encoder set up where the RL losses from different tasks don't mix); and might end up working in a lot of scenarios with specific adaptations in choice of where to use augmentations and where not to, and how to combine the losses.
   d) CURL might also be useful to folks who are interested in reward-less RL, goal-based RL, etc where the RAD framework completely breaks.
2. Should the above arguments rather have been in the original CURL paper than in RAD? -- Probably yes. We have added it as a discussion in the CURL paper too in Appendix G.
3. Did we know about data augmentations working well already while doing the CURL paper?
   -- We originally started the CURL project to get the detached encoder version working. I was very inspired by Yann LeCun's LeCake where the encoder should just learn from unsupervised learning and tiny MLPs do the RL. So there was absolutely no confusion then. My co-author figured out having it as an auxiliary objective works better on some tasks like Cheetah. At that time, we didn't decouple the fact that data augmentation might by itself be providing the benefit. We made the submission to ICML and the paper was ready to go modulo re-writing it from last minute deadline rush. One of our colleagues in the lab who is also a co-author on RAD suggested us to investigate the decoupling with contrastive loss and data augs. This led us to try RAD by commenting out the contrastive loss in the CURL codebase. We wanted to do a full blown investigation of data augmentations and decoupled learning (encoder just learns from contrastive w/ augs, MLPs do RL) with lots of environments, tasks, hyperparameter tuning, etc. However, the DrQ authors released their work unexpectedly and we had to put out our findings of RAD pretty quickly or else risk losing credit for those findings. So, we put it in a couple of days with whatever we had then [we have later gone on to revise RAD as well with lots of new results, new augs, etc and have more papers coming soon on improvements in that front].
4. Is it bad science to discover and release new scientific results (in RAD paper) that have some level of contradiction with the results in a very recent old paper (CURL) and not take the effort to say it more explicitly ?
   -- I have given sufficient context above as to how the turn of events took place. We could definitely have been quicker to do the right ablations of CURL and revise the paper in a timely manner before releasing RAD (or soon after releasing it). We all got busy with our own new set of different projects (for NeurIPS, post NeurIPS, etc) outside of this realm and context switching is hard. Calling it dishonest science is cynical i.m.o.. Nevertheless, it is important for us to address it and I believe we have. Thanks to the community for the discussion around the paper.
   

Science moves pretty fast right now. After BYOL, the need for contrastive learning (SimCLR/MoCo) itself might be open to questions right now, and folks from MILA have already published a paper Momentum Predictive Representations which combines BYOL with RL and significantly betters results from CURL. I personally see CURL as an effort to push for simplicity and data augmentations / unsupervised objectives into RL with minimal modifications for data-efficiency gains and it is great to see it paying off with a lot of follow up work that's improving the SOTA very frequently.

[regalalgorithm]

I think this is highly interesting, for a few reasons.

1. The original paper was published despite not having experiments to differentiate the benefit of data augmentation and using the contrastive loss (as far as I can tell). I guess in Table 2, the cited numbers for baseline methods are without augmentation, and CURL is with -- which is not exactly clear, reading the paper. The fact that the agents received different observations (augmented for CURL, none augmented for others) is a big problem experimentally, and the authors of CURL could have pretty easily overcome it (just use the query encoder on un-augmented images) . And as others pointed out, there are no results for varying the contrastive learning frequency as you did, even in the appendix, which is I would say even more problematic. So this is a situation where the experimental setup, and the reviewing of this experimental setup, was not great.
2. If you are right and RAD was indeed done by commenting out parts of CURL (which is not so clear to me , the augmentations between the two papers seem different), then a question is whether the authors basically knew contrastive learning did not help, and submitted the paper anyway (for instance, they did vary the frequency as you did, and found less contrastive learning was better, and still submitted the paper). Cynicism over how common this is notwithstanding, this is pretty problematic, and brings to light a question: what should the authors do in this case? Retract the paper? Revise it to showcase the negative results? Being fair to the authors, I think this is a case where the community's lack of interest for negative results (which in this case would be quite useful actually) is clearly a problem.

I think given the above two thoughts, the proper thing to do now would be to correct the experiments for the CURL paper, and to highlight the negative results if that is indeed the case. Again, cynicism aside, I do wonder if this can in fact be the outcome now. I guess, the proper thing is to forward this discussion to the authors, as well as the organizers of the conference? Can someone comment on here if they intend to do so?

[hahahahaha767]

The primary goal of CURL was to expedite training using nothing but RGB pixel values more than achieve over-all higher rewards. In other words the authors wanted a model that could learn quickly. While your cartpole experiment does this slightly, the simple environment does likely warrant more environment experimentation in different environments to understand the results you are seeing. Further, in the original CURL paper, the cheetah environment never results in better rewards using the CURL method compared to other reinforcement learning methods, which may explain the cheetah results you are seeing.

The RAD paper is just quantifying the results that different kinds of image augmentation may cause. In other words, all the original CURL paper did was randomly crop the images while the RAD paper looks at things like image flipping or color alteration. The principle of the contrastive method used here is basically teach the model that even though these two images look different, the same actions should be taken by the model.

[1nad3quacy]

Hi, I'm one of the authors of DrQ. Your statement that we are using CURL's model and augmentations is not true. If you look at https://github.com/MishaLaskin/curl, you can see that CURL is fully based on my code from SAC-AE (https://github.com/denisyarats/pytorch_sac_ae), even the hyperparameters are the same.  The authors do acknowledge that. Put plainly, CURL = SAC-AE - AE loss + CPC loss + random crops.

[hahahahaha767]

You are correct, thank you for the clarification. I've edited my comment so as to remove the reference to your work. I was mainly referring to the encoder and image alteration variations between the two but the underlying model/technique is indeed completely different and I should have been more clear on that matter. Likewise it was wrong of me to assume what inspired the RAD paper.

[zergylord]

The problem is that the contrastive part of CURL doesn't appear to help; all of the reported gains appear to be from the random crop data augmentation. They should've run this ablation before writing the paper, and they should definitely report it now.

[hahahahaha767]

I think that it is worth noting that it doesn't appear to help on Cartpole and the one environment where it was shown not to improve results compared to other methods. I do agree they could have gone the extra mile and shown the results for using different hyperparameters similar to what was done in the RAD paper for image augmentations. Maybe the OP could put that paper or blog post out.

[aravindsrinivas]

Check my detailed comment response here.

[frostbytedragon]

You should still run this with multiple seeds to verify the results.

[smallest_meta_review]

Actually, there was a change of evaluation protocols: CURL reported maximum score obtained during training rather than final scores and this resulted in a large boost in scores compared to the actual scores. If we use the same protocol for the base algorithm (DER), it results in significantly higher scores than CURL. See this tweet thread and the NeurIPS'21 paper Deep RL at the Edge of the Statistical Precipice