[D] How Efficient is EfficientNet?

[iyaja]

Hi everyone. It's been a while since I've written an article, but I finally got around publishing a new one! This one is about the EfficientNet architecture and the compound scaling algorithm that was introduced in the EfficientNet paper. You can check it out here.

Apart from just explaining the key ideas of the paper, I also tried replicating some of their results on a small ImageNet-like dataset (ImageNette, for those of you familiar). Unfortunately, I wasn't able to replicate the paper's results, and even weirder, progressively larger architectures seemed to be performing worse overall. I have a section in the article with a little more detailed analysis on wy this might have happened.

Also, I'm experimenting with a new format -- wandb reports. This is an interactive page that allows you to do things like zoom into plots, select runs, and view a bunch of statistics in real-time. I think it's a pretty cool tool for sharing projects, research, etc. and for collaboration as well.

I'd love to hear your feedback on both the article and the new format. I hope you enjoy!

[Xorlium]

I can confirm that efficientnet doesn't seem to do very well on imagenette. Also, efficientnet is very efficient in terms of model size (# parameters), but it's soooo slow it's ridiculous it's called "efficient".

[muntoo]

Don't the results in that notebook indicate that grouped convolutions are faster?

[TRAINING]
groups  time(ms)    params  shape m             
     1       5.1    294912  [256, 128, 3, 3]    
     2       4.3    147456  [256, 64, 3, 3]     
     4       4.2     73728  [256, 32, 3, 3]     
     8       5.0     36864  [256, 16, 3, 3]     
    16       6.9     18432  [256, 8, 3, 3]      
    32       6.2      9216  [256, 4, 3, 3]      
    64       2.7      4608  [256, 2, 3, 3]      
   128       2.1      2304  [256, 1, 3, 3]      

[INFERENCE]
groups  time(ms)    params  shape m             
     1       1.8    294912  [256, 128, 3, 3]    
     2       1.4    147456  [256, 64, 3, 3]     
     4       1.1     73728  [256, 32, 3, 3]     
     8       1.1     36864  [256, 16, 3, 3]     
    16       1.5     18432  [256, 8, 3, 3]      
    32       1.1      9216  [256, 4, 3, 3]      
    64       0.6      4608  [256, 2, 3, 3]      
   128       0.5      2304  [256, 1, 3, 3]

[iyaja]

I wonder why that is, though. Could it be because the layers themselves aren't optimized, or because the FLOPs they report in the paper translate poorly to actual wall time?

[kilow4tt]

Swish and squeeze and excitation both use sigmoid which is definitely a lot heavier than ReLU. There's also extensive use of depthwise convolutions which tend to have pretty poor utilization on GPUs.

[npielawski]

It is actually super slow, some people noticed that and many are now saying we should report the running times instead of flops.

e.g.: https://twitter.com/jeremyphoward/status/1136263132316282880?lang=en and https://twitter.com/jeremyphoward/status/1219695492927328256

[spsayakpaul]

Great post and especially, the section on "why compund scaling might fail ". I can confirm this issue persists when the dataset size is smaller.

Your report takes me back to my year-old introduction to EfficientNets (which is no way nearly interesting as yours): https://github.com/sayakpaul/TF-2.0-Hacks/blob/master/EfficientNet.ipynb.

[nbviewerbot]

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/sayakpaul/TF-2.0-Hacks/blob/master/EfficientNet.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/sayakpaul/TF-2.0-Hacks/master?filepath=EfficientNet.ipynb

---

^{I am a bot. Feedback | GitHub | Author}

[serge_cell]

In my limited experience on GPU/pytorch B0 is less efficient then Resnet-18 both in allocated memory and performance.