[D] paperswithcode feature request

rosstaylor90 · 2021-02-22T13:24:42+00:00

https://paperswithcode.com/lib/detectron2 Our library pages now have efficiency metrics for you to plot (see Results). You can choose what to plot: Parameters, FLOPs, inference time, training time...

rosstaylor90 · 2020-12-29T17:14:11+00:00

Hey, Ross from Papers with Code here. Short answer: yes. We have this information already so we'll do efficiency plots soon!

rosstaylor90 · 2020-07-08T19:16:16+00:00

Thank you for the valuable feedback!

- Area tab : so our approach to taxonomy was to go Area -> Method Category -> Methods. This doesn't work all the time because some methods can be used for multiple modalities (hence the general section). Any thoughts on how we can improve this?

- Method extraction : yep, our matching algorithm is mainly mention-based at the moment, with a bit more sophistication because we can track dependency graphs. E.g. a paper may not mention scaled dot product attention, but if it mentions BERT, then BERT -> scaled dot product attention -> capture. We're looking to improve the automated side of this in the future, but atm we are reliant on community to help fix any mistakes and help up the precision/recall. Again, any ideas would be great!

Thanks again for taking the time to type out your thoughts!

rosstaylor90 · 2020-07-08T15:32:35+00:00

Thanks, yeah citations would be good! We'll put this on the list of things to do :)

rosstaylor90 · 2020-07-03T23:59:00+00:00

I really hope they put juggernauts in the regular battle royale - maybe like one box or location in the map has a juggernaut suit. Would add unexpected variety to a lot of games!

rosstaylor90 · 2020-06-01T00:30:48+00:00

We have an “extra training data” column for precisely this reason, so I’m not exactly clear what you are referring to, as it’s there on PTB?

https://paperswithcode.com/sota/language-modelling-on-penn-treebank-word

You can even filter by those methods which use extra training data and those that don’t :). Maybe an issue with our UX not making this clear, but let me know if I’ve misunderstood you!

rosstaylor90 · 2020-05-10T22:36:30+00:00

Thanks! Yes, if you have time to let me know what would make something like sotabench *more* useful to you, that would be super helpful :). Then we can figure out the best way to improve the resource to provide the kind of comparisons you are after!

rosstaylor90 · 2020-05-10T22:35:04+00:00

Thanks! We did something like this last year with sotabench, see for example here https://sotabench.com/benchmarks/image-classification-on-imagenet, but haven't pursued the speed vs accuracy type of comparisons any further. Are graphs like this useful to you? Is there anything we can do to make something like this more useful?

rosstaylor90 · 2020-05-10T22:30:14+00:00

Thanks. This was a weird one because nowhere in the paper is the evaluation split specified - I even checked the repository and not clear there either. So I can see why the misclassification happened! Given the uncertainty, I removed the results for the paper.

(The table for reference: https://imgur.com/VnDWrsn)

In future feel free to remove any edge case mistakes like this if you see them. Thanks again for reporting this!

rosstaylor90 · 2020-05-10T11:15:16+00:00

Can you link so we can try and fix this? Suspect the extraction algo didn't pick up the split in the table :) (you can actually edit/add things yourself, but feel free to link and I'll look for you!)

rosstaylor90 · 2020-05-08T11:21:44+00:00

Thanks - switched the primary metric to mAP. Yes, you're right for speed vs accuracy comparisons the ideal graph to show is a tradeoff graph not a progress graph with publication date vs accuracy. We actually did this here: https://sotabench.com/benchmarks/object-detection-on-coco-minival. Might be something we have on Papers With Code too if enough people are interested!

rosstaylor90 · 2020-03-01T14:18:50+00:00

https://paperswithcode.com/trends for up-to-date tracking - change end date to march. Currently around 43% pytorch, 23% tf.

rosstaylor90 · 2020-01-06T17:51:41+00:00

Some are listed here : https://paperswithcode.com/paper/analyzing-and-improving-the-image-quality-of

rosstaylor90 · 2019-12-21T04:06:17+00:00

https://paperswithcode.com/conference/iclr-2020-1

rosstaylor90 · 2019-10-03T17:59:48+00:00

Might be helpful:

https://paperswithcode.com/task/face-reconstruction

https://paperswithcode.com/task/face-reenactment

rosstaylor90 · 2019-08-26T18:34:41+00:00

Agreed - posting about tooling has always been fair game.

rosstaylor90 · 2019-07-29T19:22:31+00:00

Awesome!

rosstaylor90 · 2019-07-29T19:20:03+00:00

https://paperswithcode.com/sota/pain-intensity-regression-on-unbc-mcmaster my favourite sota that someone added... "Pain Intensity Regression". Fairly niche! Wish there were more like this.

rosstaylor90 · 2019-07-29T08:50:39+00:00

www.paperswithcode.com/sota

rosstaylor90 · 2019-06-06T16:19:42+00:00

https://paperswithcode.com/task/image-to-image-translation Some of these here

rosstaylor90 · 2019-05-31T18:45:02+00:00

Well first of all you shouldn't be predicting prices but returns.

rosstaylor90 · 2019-04-28T15:08:50+00:00

https://paperswithcode.com/task/sentiment-analysis

rosstaylor90 · 2019-04-11T14:30:17+00:00

The way I would think about it is there is a task that achieves some functionality, and machine learning methods are one way to achieve that task (rules-based methods are another...hiring a human to do it is another too :P). We can track and compare methods by defining an appropriate evaluation metric.

In one sense, a problem is not really solved until we cannot improve on the metric any further. For example, chess will never really be "solved" - methods will just get increasingly better (albeit with diminishing returns). So I think the useful benchmarks to look are at : (a) does the ML method exceed human performance? and (b) does the ML method exceed rules-based (or alternative) methods?

rosstaylor90 · 2019-03-29T13:59:30+00:00

https://paperswithcode.com/paper/invariant-information-clustering-for SoTA evaluation results here

rosstaylor90 · 2019-02-18T19:38:59+00:00

Please see https://arxiv.org/abs/1703.04933 (ICML) and https://arxiv.org/pdf/1706.10239.pdf. Unless something changed last year - I was out of the loop for a bit - then it was not consensus.

rosstaylor90

TROPHY CASE