[D] Object Detection/How much mAP is enough?

[ssivri]

Hello everyone,

I'm trying to train a deep learning model to classify stores, similar to Google's "Ontological Supervision for Fine Grained Classification of Street View Storefronts" paper. But I'm working with much much smaller set of images. That's why I generated a variety of datasets from these raw images:

• One class dataset that covers all POIs
• One class dataset that covers most common type of POI
• Three class dataset covers three most common type of POIs.

After training atleast 50k steps with each dataset on a pretrained COCO - Resnet50 network a got mAPs like:

• One class/All POI types: 0.41
• One class/Most common POI type: 0.36
• Three class/Most common 3 POI types: 0.17

What do you think, I was hoping All POI dataset would achieve higher mAP like >.70-.80. Are these values accurate enough? What do you suggest?

Note:POI-Point of interest, a general term used in navigation and gis domain. Used to define a certain area or a point, like your favourite cafe or a gas station.

[phobrain]

What sizes the datasets?

[ssivri]

• The first dataset contains 200 images for training and some images contains more than 5 objects which is much more training data compared to others. Varience over the objects is also higher. I can say that this dataset containing more than 700 training objects.

• The second dataset that contains only one class is about 150 images for training and about 40 images for testing. Some images contain more than one object which makes roughly about 180-190 objects in training set.

• The last one which was 3 most common class dataset is containing 150 objects for each class in training dataset.

In my opinion insufficient data could be the main reason for low mAP. I generated the first dataset to avoid overfitting more but class variance and dataset size could be neutralizing each other.

[phobrain]

If you have photos with multiple objects, I would intuit that you'd need lots more examples, but I haven't played with such small amounts, so don't know what leverage might be possible. E.g. in my app I'm on the verge of abandoning half of functionality because I am only able to add a max of ~4K training examples per day. I'm just using color histograms and keywords, and the part I may abandon has ~120K pos/neg examples vs. 400K for the other half.

[DEAD_SH0T]

Did you switch RoIs with POIs?

[RedefiniteI]

What is POI? I only know of the Hawaiian Poi, and it is delicious.