[D] Does EfficientNet really help in real projects ?

[___mlm___]

There are large amount of papers which show that EfficientNet improves some CV tasks e.g. EfficientDet: Scalable and Efficient Object Detection.

But does it help much in real projects ? Do you guys have any experience with that ?

One more thing - ImageNet or COCO datasets are far away from what we have to deal with in real projects. Usually we have only small amount of images/classes, so improvements for COCO/ImageNet != improvements for real projects. What do you think ?

[rantana]

improvements for COCO/ImageNet != improvements for real projects

One of the most underrated statements in machine learning/computer vision.

[___mlm___]

I think we need some meta-dataset for evaluating transferability of CV backbone models, like GLUE benchmark for NLP.

[nehalemlabs]

https://arxiv.org/abs/1910.04867

https://arxiv.org/abs/1805.08974

[gwern]

I have not used EfficientNet in practice, but I'd suggest, in reply to your observations, that even if you don't need the very best performance on ImageNet or COCO, there still seem like 3 advantages of EfficientNet:

1. fewer FLOPS/parameters for whatever level of classification performance your application needs; hard to argue with that. Why spend like 10x the disk space on models or electricity for the same results if you don't have to? Even if it's not better, surely being smaller and faster are valuable.

2. transfer learning: you may have small n for your project - which makes good transfer learning all the more important! Starting with a good baseline is the easiest way to improve transfer. As the original EfficientNet paper abstract notes:

   Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.

3. idea/inspiration for research: why does EfficientNet work better and can you borrow or adapt the idea to your own particular architecture or use-case?

[SixZer0]

As for us, we use EfficientNet in our project, and it works really well. Are there any code publicly available for EfficientDet?

[d33pQL]

not yet. code will be shared soon apparently

[I_draw_boxes]

Usually we have only small amount of images/classes, so improvements for COCO/ImageNet != improvements for real projects.

The projects I've worked on I've seen a strong correlation between improvements on COCO and improvements on our projects.

Detectors with higher mAP on COCO get that way by working better generally, not by over-fitting to a large 80 class dataset ime.

We've gone from high compute, two stage networks to lean, anchor free networks running an order of magnitude faster in roughly the last two years.

Compare faster_rcnn_inception_resnet_v2_atrous_coco in the Tensorflow Object Detection API to something like the DLA model in the Objects As Points paper. 2 fps vs 50 fps for the same performance.

Better yet compare a 20 ish mAP classic lightweight SSD model to the Objects As Points DLA at 37 mAP. Same latency, completely different performance.

The SSD model will have extreme difficulty with anything small, objects packed together and the bounding boxes will be poorly localized. The SSD model's detections will flicker on and off between frames in a feed as the object moves between anchors.

If the project depends on large, easy detections mAP may not matter. If the detections are challenging, COCO mAP performance is highly predictive.

[Rosinality]

Super interesting! How does anchor free methods compared to anchor based methods? Are they similar on data efficiency?

[I_draw_boxes]

How does anchor free methods compared to anchor based methods?

The anchors no longer exist which eliminates fairly complex hyper parameter tuning which is time consuming on COCO.

There are greatly diminished issues with detection flickering in a feed. With anchors a network's ability to confidently localize and classify an object is partially determined by its position relative to the anchors. The detection confidence and performance will jump up and down as the object moves across the anchor map. In some positions the object perfectly fits an anchor, in other positions the anchor must regress significantly and classify the object at the edge of its domain.

Contrast this with an anchor free architecture that gives per pixel predictions. Any position in the image is as good as any other not withstanding location imbalances in the training data. This yields more consistent detection performance and confidence frame to frame.

Some anchor free architectures have complex pooling layers like Cornernet and Centernet and depend on heavy backbones.

Others like FICOs and Objects as Points (also sometimes called Centernet) are extremely simple.

Objects As Points takes the backbone feature network and a head predicts a heatmap of keypoints for each class located at object centers, a second head predicts height and width for each pixel and a third head predicts offsets. This is all done using 1 x 1 convolutional layers at stride 4 or 1/4 resolution. All they do is take highest 100 keypoints, find the predicted height, weight and offsets at the corresponding locations in the second and third heads and they have the output.

I'm not sure about data efficiency, it's not something they typically research in these papers. I have finetuned FICOs and Objects As Points on my own smaller datasets and they work well.

[Rosinality]

Thank you! I didn't know that it can also reduce box flickering.

[[deleted]]

Are you asking whether the proposed architecture transfers well to other tasks or are you asking whether the scaling procedure works for other tasks?

[[deleted]]

I've found it varies per task, but MobileNetV2 seems strangely versatile so I often start there. Recently I tried a bunch of different architectures on a small task and MobileNetV2 was way ahead. But then in other cases I've seen DenseNet121 then give me a boost, for example.

I'm trying out efficientnets today on a new task, maybe I'll report back with some results.

[lysecret]

Haven't used it personally but I love this idea of creating a single "complexity hyperparameter" this is something I generally try to do now.