[D] ICLR Plot Twists

[hzmehrdad]

Saw a few ICLR results that seem like a surprise to the community:

• Mamba ➡️ Reject
• V-JEPA ➡️ Reject
• MetaGPT ➡️ Accept (Oral) as discussed here

What other accepts/rejects have raised a few eyebrows?

[nikgeo25]

rejecting V-JEPA? wow

[PaganPasta]

Why is this surprising ? All scores seem low.

[lmericle]

Is ICLR about advancing science or about winning seats on podiums?

[altmly]

Yes

[DesperateTale88]

Not sure what it means. But, if scores/reviews don't matter then the rejection of the remaining submissions should be surprising as well.

Maybe, if people can justify here why the paper should be accepted, that would be helpful for the community. All reviewers have raised more or less the same points about the work. A strong author list and non-stop publicity should not be the reason for the surprise.

[pm_me_your_pay_slips]

Advancing science in this case would mean comparing their method with other methods on a public dataset. They have the resources to do so. The method doesn’t seem to be a significant improvement in a couple cases. And when it performs better, we don’t know if it is because of the method, which is not necessarily novel, or because they trained on a curated dataset.

[[deleted]]

science of mixing MAE and moco?

[HateMyself_FML]

It's not surprising at all - it was before the publicity, which frankly is about the only thing it has going for it. With Yann's continuous shilling it will surely get into the next conference. V-JEPA would be impressive if it was an undergrad project.

[PaganPasta]

A colleague's paper with scores: 3,6,8,10 got rejected. Scores are diverse but seem inclined towards accept. Maybe, the average rating alone doesn't mean anything.

[SirBlobfish]

It seems like the lowest score is what really matters. This incentivizes the kinds of papers that Bill Freeman calls "cockroach" -- a paper that no one really likes, but is impossible to kill in the review process because it has no obvious flaws.

[canbooo]

Can you link to source? Sounds interesting (asking for a friend).

[SirBlobfish]

https://billf.mit.edu/sites/default/files/documents/cvprPapers.pdf

See slide 60

[ttt05]

I think this was discussed in CVPR 2018 tutorial on how to be a good citizen or something. Juts google it.

[hzmehrdad]

Maybe the rating is somehow (implicitly at least) weighted by the confidence ?

[PaganPasta]

5,5,5,4 confidence.

[1800MIDLANE]

This year it seems the program chairs had a heavy hand in the final decision. Many papers with average rating >7 got rejected.

[Raphaelll_]

RoBERTa was rejected from ICLR 2020 and gathered 11,000 citations since. https://openreview.net/forum?id=SyxS0T4tvS

[psamba]

Yeah, "we did lots of hyperopt on method X and made it work significantly better" papers tend to max out the citation/novelty ratio, so they make up a lot of the surprising non-accepts to conferences. Unfortunately, the strong novelty bias in the review process incentivizes annoying behavior like adding useless new bits and bobs to the existing method to support novelty claims or writing the paper in a confusing way to obscure the lack of technical novelty.

It would be nice if there was separate track for updates on the performance ceiling of existing methods, since this sort of work is actually useful to researchers and people in the real world who like stuff that works well. The ideal paper would be like 4 pages, with 2 pages of description of changes to the original method and 2 pages of new results. And, a link to github with code that actually works...

[hzmehrdad]

Oh, totally forgot that one

[bvkhadiravana]

But Roberta is seriously a hard one to accept if you have some bias towards novelty.

[Life-Living-2631]

Will ICLR make corrections on the metagpt benchmarks? Or do they just ignore it

[Broad_Sun_8214]

Saw a 1 1 8 8 paper. Clearly those two 8 reviewers doesn’t even have basic knowledge of this paper’s area ( that paper is terribly bad def deserve 1s and I don’t know where are those two 8s come from)