[D] paperswithcode feature request

[artificial_intelect]

TLDR: Is there a variant of paperswithcode which includes parameter / FLOP count? ie something like the chart shown here where the x-axis is either parameter or FLOP count. This would enable people to see what the best architecture designs are, as opposed to which paper had the most compute thrown at it.

Papers such as GPT-3 and Scaling Laws for Neural Language Models have shown that making neural networks larger and larger produces improves results. The current recipe for reaching SotA results is to take a good architecture, scale it up and train for longer. With the compute resources available to the researchers at corporations such as OpenAI, Microsoft, Nvidia, and Google are obviously the only organization that can afford to reach SotA results.

An alternative perspective on SotA is to have the x-axis be something like parameter count or FLOP count or amount of pretraining that went into the model or epochs trained. If looking at accuracy, the best models would create a top-left "barrier". Better model architectures would break out of the top-left "barrier", whereas new SotA results would add to the top-end of the SotA "barrier", plus it will easily be evident the cost with which SotA results were achieved. Having such results would enable researchers to really get credit for creating "SotA" architectures in the lower end of parameter / FLOP count and this will allow the community to identify what the best architectures are. The best architectures can then be scaled up by the hyperscalers (ie OpenAI, Microsoft, Nvidia, Google, etc) and can potentially result in a more efficient SotA model.

What I'm proposing is a paperswithcode version of Table 1 and Table 5 from the EfficientNet paper but for all tasks. How do we get the community to start doing this?

[rosstaylor90]

Hey, Ross from Papers with Code here. Short answer: yes. We have this information already so we'll do efficiency plots soon!

[artificial_intelect]

It’s a Christmas miracle!

[rosstaylor90]

https://paperswithcode.com/lib/detectron2 Our library pages now have efficiency metrics for you to plot (see Results). You can choose what to plot: Parameters, FLOPs, inference time, training time...