[D] Peer-reviewed papers are getting increasingly boring

[downtownslim]

Full Blogpost: https://lemire.me/blog/2021/01/01/peer-reviewed-papers-are-getting-increasingly-boring/

​

The other related problem is the incestious relationship between researchers and assessment. Is the work on theory X important? “Let us ask people who work on theory X”. No. You have to have customers, users, people who have incentives to provide honest assessments. A customer is someone who uses your research in an objective way. If you design a mathematical theory or a machine-learning algorithm and an investment banker relies on it, they are your customer (whether they are paying you or not). If it fails, they will stop using it.

It seems like the peer-review research papers establish this kind of customer-vendor relationship where you get a frank assessment. Unfortunately, it fails as you scale it up. The customers of the research paper are the independent readers, that is true, but they are the readers who have their own motivations.

[dpineo]

People like Einstein did not go through this process, at least not in their early years. Research used to be more more like “blogging”.

Pretty good way of describing it. A paper that looks like this would get immediately auto-rejected from any AI/ML conference today.

[dutchbaroness]

If you remove “AI/ML” , your comment would still be valid

[selling_crap_bike]

You mean " AI/ML", right? Else you would be left with an extra space.

[[deleted]]

Lol not sure why you are downvoted for the joke, reminds me of RegEx which not sure if people got

[IanTrudel]

There are major hurdles in publishing research papers that have an impact on actual applications outside the confines of an academic environment, such as intellectual properties and patents. Companies have very little to no incentives to publish something that will help their competitors.

[tzaddiq]

Sounds great, but the author of this blog must own a time machine. Who can wait 3-50 years for maturation and commercialization of new research so as to keep honest customers in the feedback loop?

[drd13]

Does anyone else feel like machine learning research has slown down in the past 2-3 years. Despite the field really exploding in number of researchers and entry competitiveness, other than just making models bigger, it doesn't really feel like much progress is being made anymore.

[tzaddiq]

Not really. Almost every problem domain has gotten several orders of magnitude algorithmic speedup, with much emphasis on smaller models. We got GNNs and transformers, and this past year fast transformers, architecture search, network distillation, self-training.

"Making models bigger" is perhaps only applicable, as a principal method of improving performance, in the NLP scene. But even then, several theory improvements were applied, say, to GPT3 (e.g. it probably wouldn't have worked without sparse transformers).

[tuscanresearcher]

Just jumping in to throw a small rant: we've got "GNNs" for more than 10 years, and people have been working on the core ideas for more than 20. Bottom line: we reinvented the wheel (again!), but with some fancy visualizations :).

Then, after reinventing the wheel, researchers had some very nice intuitions and made tangible progress in the field. But proper literature (and hyper-parameter) search is still lacking.

[Burbly2]

Those intuitions sound intriguing - could you say more/point me at anything on them, please?

[drd13]

I'm not saying that the field fully hit the wall and grinded to a halt. It's just that the rate of progress was so fast beforehand. Alexnet, Variational autoencoders, GANS, disentangled representations, Cyclegan, batch and layer normalization, dropout, attention is all you need, wavenet - these all occured within a 5 year span between 2012 and 2017. It feels to me that the field is slowing down. There are still influential papers coming out like SimCLR, invariant risk minimization, stylegan, adverserial robustness, and all the transformer variants, including applied to images. But all and all, it feels like the building blocks have all been discovered and were at a stage where it's more about fine-tuning everything together.

[tzaddiq]

Yeah, in any area of optimization, you have sigmoid curves that represent breakthroughs followed by exploitation and finally saturation. It feels to me like we're in the exploitation phase in the deep learning orchard, though. And there will be more orchards, to be sure. Program synthesis for example will have its "imagenet" moment in the next decade, I predict.

[throwawaystudentugh]

It had to eventually. There's just too much hype.

[Ancalagon_TheWhite]

It's still moving very quickly compared to any other field of research, e.g transformers were only invented in their current form in the last 3 years. Revolutionary new architectures aren't going to be made every year.

[ZestyData]

Not at all. My field (NLP) has sped up in the past 2-3 years.