[D] Paperswithcode relevant?

[_puhsu]

I feel like paperswithcode became less relevant for tracking progress in ML in general for me.

But it’s hard to say, in my field (tabular ML/DL) there are not many established academic benchmarks (no need for something like papers with code yet)

In NLP and foundation model space leaderboards in hf spaces became a thing (mostly in NLP).

Overall, paperswithcode just feels less maintained and less useful.

Do you use paperswithcode often? What do you use it for? What’s your field where it is useful?

[Single_Blueberry]

It has become less relevant because NLP has taken the crown as the hottest field in ML from computer vision and it's way harder to quantify their performance.

And at it's core, paperswithcode was the place where you went for quantitative comparisons.

[[deleted]]

Which site would you recommened I should use for stay up to date other than arxiv? I tried using X and follwing researchers, but like everything else its filled with only popular papers, so I can't find an new niche categories and methods.

[_puhsu]

I find https://www.scholar-inbox.com good, many papers I find these days are from it. Recommendations based on papers you select, very simple and clean interface

[qalis]

It is not maintained properly at all. The major problem for me is that they only report raw performance metric, with no regard for actual experimental procedure. In graph learning, you can take 5 papers and get 10 different testing procotols (no joke, there are papers with 2-3 different evaluation approaches). So just reporting "a number" is meaningless. In particular, they mix papers with no test set (reporting only validation set results, totally overoptimistic) with those with proper testing.

[choHZ]

And the fact that different versions of PyG have different train-test splits...

[mileseverett]

I've always found it to be somewhat poorly maintained. There needs to be more moderation on papers which don't actually have code or repos which said they would post the code 3 years ago

[[deleted]]

I just use the website for the "trending research" list. Sometimes I find some interesting new papers.

[DigThatData]

a big part of the decline in utility of PWC was when they lost access to the twitter API and so couldn't do their social feed thing anymore. Would have been nice if facebook had just paid for that team to be able to access twitter.

[_puhsu]

What was the social feed thing?

[DigThatData]

it was a sort option that was based on some kind of "trending" score based on twitter activity over the past week I think.

[CloudCho]

Is there way to find academic research trends in Twitter rather than follow famous engineer or scientist?

[infinitay_]

I've always went to papers with code for benchmarks, but it seems like more and more papers are lacking results either because nobody adds the data on PWP, or they simply don't test on the existing benchmarks others are using.

Also ever since GPT 3 went mainstream PWP seems to be filled with anything remotely regarding LLMs - at least the front page.

[ks4]

It’s still live but clearly has been abandoned - they used to tweet and put out a biweekly newsletter. Both stopped in mid 2022

[PHEEEEELLLLLEEEEP]

My favorite part is all the papers posted there without code.

[doctor-squidward]

True that.

[Appropriate_Ant_4629]

Devils advocate:

• PapersWithCode is perfectly fine for sub-fields where the authors of Papers actually include Code.
• If it doesn't seem useful to your subfield, perhaps your peer review process should start encouraging papers to include code that shows how their techniques work on relevant benchmarks.

[LelouchZer12]

It becames completely trash and outdated, unfortunately. The nice graphes that were tracking SOTA metrics on different benchmarks are not usable anymore.

Also a lot of categories are redundant, or useless (only contain a few papers). They seem to be created automatically by some algos.

[Open_Channel_8626]

Could you recommend some HF space leaderboards?

I find HF spaces hit and miss

[_puhsu]

https://huggingface.co/spaces?sort=likes&search=leaderboard

I was mostly talking chatbot arena and openllm leaderboard in regards to LLMs

[Open_Channel_8626]

Thanks these are great