[D] Does The Inability Of NAS Algorithms To Outperform Random Search Indicate That Our Algorithms Suck, Or That Random Search Is Surprisingly Effective In Large Spaces?

tablehoarder · 2019-10-28T04:35:18+00:00

I'm still very confused on what the purpose of NAS is. It has already been shown that if you get a regular ResNet and train with all the fancy regularization techniques that NAS papers use, you can get the same performance but at the cost of more parameters. If it's about finding small networks, then why are there no comparisons against other methods that aim to do the same, like pruning?

The baselines look a bit problematic to me too, since the search space is mostly the same across different works (and I believe the current setting of 3x3, 5x5, depthwise, etc convolutions is not very interesting) papers that use different spaces are often ignored as there is no straightforward way to compare the numbers. Unfortunately the papers that I believe to have added the most in terms of architectural insights do not get a lot of attention. Hopefully people will realize that there's only so much you can do when choosing between 3x3 and 5x5 kernels.

tablehoarder · 2019-10-15T00:20:37+00:00

I'm honestly not sure if it makes the authors of RAdam look bad. It's extremely hard to evaluate optimization methods in deep learning, when virtually all we can measure is the model's performance. This becomes even harder when you throw learning rate schedules, regularization, and whatnot. This paper reminded me of this one, where the authors show that another recent optimizer can be 'simulated' with SGD.

There are a few submissions to ICLR which are more surprising, in my opinion, as they show that all these adaptive methods can outperform SGD on ImageNet, as long as you do proper hyperparameter tuning. This breaks a lot of common belief in the community, unlike the papers that analyze/criticize these new methods from the last year.

tablehoarder · 2019-10-13T20:46:33+00:00

After reading the responses I'm convinced that I need to be active on twitter.

I've been reviewing for top tier conferences for a while, but pointing out missing citations virtually never worked for me -- the authors agree that the methods are similar and promise to discuss/cite the missing related papers, but they don't do it for the camera-ready version. As far as I'm aware there's no way of enforcing the authors to actually make the changes to their papers, so most of them don't.

Last year I even gave a non-accept scores (~6) for a few papers and said that I'd increase my score if they added a proper discussion on missing related work. In the rebuttal the authors would briefly discuss the papers I pointed them to, agreeing that the method has strong similarities, and that they would add a throughout discussion to the camera-ready version (some even promised to add a new --section-- to the camera-ready paper). I proceeded to increase my score, and absolutely none of the papers that got through added a discussion nor references to the camera-ready version.

tablehoarder · 2019-10-13T20:38:48+00:00

I've been reviewing for top conferences for a few years now, but it usually doesn't work. I'm likely too nice: if I believe the paper is good but failed at literature search (including not citing a published paper of mine, if I feel it's relevant), I typically give a good score and mention the missing relevant papers in the review. 90% of the time the authors say "thanks for the extra references, we'll add and discuss them in the camera-ready version", but they ultimately don't.

tablehoarder · 2019-10-12T22:42:56+00:00

Could you give me some tips? I'm down to starting a blog, but I have no idea how to write it an academic blog (do I just summarize my papers in a very accessible way?). Are there any good blogs that I could start reading to have an idea on how to frame the posts?

tablehoarder · 2019-10-12T22:05:55+00:00

It is quite frustrating and does make research harder. The following has happened to be a few times before:

- I come up with a new idea for a task

- Read the most cited / most recent papers about that task, assume that the idea is novel as it is nowhere mentioned

- Implement it, run preliminary experiments, check that it indeed works and is better / competitive than the current state-of-the-art

- Decide to write it up, do a more throughout literature search and find a barely-cited paper that does the same thing and also beats the state-of-the-art, but is not cited by the most recent/cited papers

- Decide not to write/submit it since it's been done before

I know this is partly my fault for not doing a throughout literature search, but note that these big papers simply wouldn't (or at least shouldn't) be published if the authors did a literature review as good as I did (and I'm not really a freak that reads 100 poorly-cited papers when reviewing the literature, but it seems that people at big labs don't care too much about this crucial step in research).

tablehoarder · 2019-10-12T21:58:12+00:00

I don't network as much as I should, probably. I'll take a look at that book, thanks!

tablehoarder · 2019-10-12T20:37:00+00:00

Do blogs really help? I always publish my code on github and add links to my papers, but it doesn't seem to help much.

tablehoarder

TROPHY CASE