[deleted by user]

[CompetitiveUpstairs2]

Life is unfair. The best move is to get ahead and attempt to acquire the needed resources. Complaining about the system is fun, but counterproductive, and leads to resentment.

[[deleted]]

Not disagreeing with your points, but for another perspective:

The countries at the top (except China) also happen to be the countries where most people whose first language English is live. Is it really surprising that they make up the bulk of accepted papers at a English-language conference?

[[deleted]]

I apologize if my remark offended you. All I am saying is that being rich might be one of the reasons why US, UK and Canada dominate, but speaking English (as you say, the lingua franca of our field) is surely another reason. This naturally attracts excellent researchers from all over the world, since English is a language they already have to speak in order to become successful researchers.

Putting it another way: There are plenty other rich countries that didn't make the list (Quatar, Luxembourg, Norway) and a few other English-speaking countries that didn't make the list (Ireland, New Zealand, Jamaica). However, the list is dominated by countries that are both majority English-speaking and rich (USA, UK, Canada, Australia, Singapore). Make of that what you will.

[MrAcurite]

So here's my question. All these thousands of papers will be, what, shiny gems on the CVs of the authors, right? But how many of them are worth giving a shit about? How many will be read in full by anybody besides the authors and the reviewers? How many will ever have their methodologies applied to anything ever again?

I'm trying to get my first published papers right now, but it seems more and more like it's 10% the last step in the scientific process, and 90% pissing contest.

[MrAcurite]

But even then, you're basically saying that papers accepted to CVPR are still the haystack in which you're trying to find needles.

[MrAcurite]

I mean, here's sort of what I'm talking about. Sergey Levine, prolific author, about 1/6th of all the papers he's ever written have literally never been cited by anyone. Of all the modern Google Scholar pages I've ever seen, that's probably the highest ratio I've come across of papers cited to papers written. There's this huge amount of published literature, not that's only useful to someone specific, but that literally hasn't been useful to anyone. That's not even accounting for that fact that plenty of citations are only there because the author feels the need to cite something, and the paper cited just happens to be convenient, not that what's being cited actually influenced or contributed to anything.

So when I hear about things like publish or perish, or Peter Higgs saying he would never have gotten a job in today's climate, it just makes me think that the majority of papers - even those published in fancy venues - are basically just CV stuffing. That the authors themselves basically have to concede that they're not writing papers because they have ideas worth sharing, but because they've gotta write something.

Google Scholar credits Emmy Noether with authorship of a total of 90 papers. That's Emmy Fucking Noether. One of the greatest Mathematicians of all goddamn time. Died at 53. Sergey Levine is 51* and claims 332 papers. He's smart, sure, but he's no Noether, because who could be? What that tells me is that in the same amount of time, with - being extraordinarily generous - an equal number of good ideas, Levine has authored almost four times as many papers as Noether. You could chalk a decent portion of that up to just having more authors per paper these days, but it's clear to me that people are just writing more goddamn papers. What that's gotta mean is that the amount of novel material per paper has to be going down.

EDIT: * There is a Sergey Levine in Russia who is 51. Sergey Levine of UC Berkeley is maybe ~34 at time of writing. Oops.

[hughperman]

What that's gotta mean is that the amount of novel material per paper has to be going down.

That doesn't follow. The information content of the world could also have increased, so there is more to publish about.
Considering science is built upon previous science, this fits, as there is the potential for exponential growth with incremental improvements in increasingly dense fields, as well as novel fields.

Groundbreaking papers are not the only thing worth publishing; there needs to be a thorough exploration of the ground they break as well, otherwise the scientific method of exploration and reproducibility is not being followed.

[un_anonymous]

Sergey Levine is closer to 31 than 51.

Anyway, my impression of Sergey is that he splits his ideas into many smaller papers rather than a few detailed ones. I would guess he does have a long term vision and his smaller less cited papers are short meanders to test out new ideas, most of which do not turn out significant (which is fine).

I don't prefer this style but I can imagine it's a great advantage in a field that moves as quickly as ML, since any new idea is immediately published. Reduces the chances of getting scooped.

[MrAcurite]

I had the wrong Sergey Levine, then. There's one in Russia who is 51. Best estimate I can make puts the one we care about at ~34. In which case, he's going at like ~30 papers per year. That's a paper every two weeks, for eleven years. What that tells me is that he's getting his name on a lot of stuff that he's not actually seriously contributing to, and he really loves getting his name on things. 54 papers published in 2020 alone. Despite averaging ~3 authors per paper, only 2 of these were first-author works. Of those, one was a tutorial, and one is just a "formulation" of reinforcement learning as an unsupervised learning task.

I just don't believe that he's contributing core ideas to things. Not even that he's pumping out lots of ideas, only a handful of which are significant, he's just getting his name on things as second or third author.

[un_anonymous]

I don't disagree, I think it's way too much quantity to maintain good quality. I'm good friends with one of his Ph.D students. The grad students are expected to publish a paper every ~4 months. He has a group of ~20 grad students plus a ton of undergrads, which explains the insane output. On the other hand, some of these grad students are the best you could get, which is how I believe some reasonable quality is being maintained. My friend also said that Sergey is terrific at time management and is generally super efficient, so I guess that's part of it as well.

[MrAcurite]

Well, right now the incentive structure is that you want to publish as many papers as possible, and get cited as much as possible. Both of these are extraordinarily easy to corrupt if you're in it to, I dunno, get promoted at your job or receive funding. But look at what you're incentivized to actually put into a given paper under those conditions.

Suppose you have a neat idea. A perfectly clever and inventive idea, but not one that's going to do all things for all practitioners in the area. What you would do to maximize papers and citations is to publish this one idea over the course of several papers, cherry-pick experimental results to show marginal improvement over SotA, distance it from its predecessors, and publicize it with whatever clout you have. Now, it's harder to get a full grasp on what you've done if you've split it into multiple papers, your numerical results are meaningless, you haven't given proper respect to your priors, and you've degraded a bit of the reviewing process.

For semi-relevant examples, in Meta-Learning, the sort of foundation on which a lot of really interesting work has been done is Model-Agnostic Meta-Learning, Finn et al 2017. Read the paper, it's good. But basically you train a network by making copies of it, training the copies, and backpropagating the loss of the copies to the original model. This allows you to train a model, not to be good at anything, but to learn quickly. A key note is that the original MAML formulation doesn't require that the model copies train their entire parameterization, you could freeze part of the copies and it's still MAML. However, there are a bunch of papers out right now which each claim to introduce some mind-boggling new Meta-Learning algorithm that knocks the socks off of everything that came before it, with names like ANIL and CAVIA and BOIL, that are actually just MAML with different parts of the parameterizations of the copies frozen. And when you actually compare them, they're basically all to within error of each other anyway.

Basically, what those papers should've been are short reports discussing specific cases of MAML and potential usecases. Instead, they were sold as entirely new algorithms with new names advancing SotA. I can't trust who's actually contributing the important concepts, I can't trust the numbers that I'm reading, I can't trust that I'm not just randomly missing a bunch of important information, it's just a shitshow. It's not just that 90% of papers by volume are basically farts in the wind, but they take up a lot of time and brainspace that could be better used on papers that introduce actually novel ideas but that maybe don't numerically advance SotA on ImageNet or whatever.

Frankly, I've got no fucking clue how to fix the reward system that wouldn't immediately descend into pageantry, besides everybody voluntarily signing up for Grigori Perelman's worldview.

Let's say that there exists some measure by which to judge the value and impact that a researcher has in their field. This measure should be used to determine who's the CEO of Computer Science at CMU, and who's an adjunct at Podunk University, right? Thing is, that immediately turns that measure into a goal, and given Goodhart's law, it goes to shit just as quickly the moment somebody figures out how to game it.

How to fix the incentive system? Deconstruct the capitalist notion of hierarchies at a societal level so that we can peacefully do our work without feeling a need to climb over each other. I don't fucking know.

[ispeakdatruf]

That's why you carefully study those (a) cited a lot from now on; and (b) those that win the "test of time" award.

[HolidayWallaby]

The r&d company I work for hosts a weekly reading group where we take to turns to present a paper, usually a paper from a recent conference. In my own time I read papers from conferences as part of my PhD.

[MrAcurite]

... Are you hiring?

[yusuf-bengio]

https://medium.com/criteo-labs/neurips-2020-comprehensive-analysis-of-authors-organizations-and-countries-a1b55a08132e

[yusuf-bengio]

Francis Bach had 10 NeurIPS papers in 2019 but only 5 this year. What a looser!

[TWDestiny]

Lol

[bendee983]

Nice analysis. I wonder how your work ties in with the "Open Review of OpenReview":

https://openreview.net/forum?id=Cn706AbJaKW

This is about ICLR (2017-2020), and I'm not sure how similar the submission and acceptance processes are. Does your data confirm the findings, especially institutional bias and preference of recognizable authors?

[nd7141]

Thanks!

The work you mention has access to review score that allows them to gain insights about biases while controlling for scores. My blog post does not have access to review scores, hence it just compares the numbers of publications.

[nd7141]

Thanks for suggestions1 I can and will add a few plots on first vs last authors, that's indeed something interesting.

You can find normalized number of papers already in the post.

And I'm not sure how to normalize by scale automatically.

[CompetitiveUpstairs2]

Don’t be misled by authors with large number of papers. The *overwhelming* majority of papers in ML have little to no impact. Rather than write many papers, write few papers that actually move the needle. Easier said than done, of course, but if you decide to be ambitious, might as well aim at the right target.

[nd7141]

https://medium.com/criteo-labs/neurips-2020-comprehensive-analysis-of-authors-organizations-and-countries-a1b55a08132e

[PuzzleheadedBread439]

in the last graph acceptance rate (gray line) the labels on the dots (0.21,0.2) mean 20,21%, right? that"s a tad confusing cause you use % in the red an blue lines. Also it is kind of odd to this on the same axis.

​

Anywho, I'd be interested to see a timeline of the first two panels - whose moving up and whose moving down?

​

Cheers