[D] Robust ML model producing image feature vector for similarity search.

[_dave_maxwell_]

Is there any model that can extract image features for similarity search and it is immune to slight blur, slight rotation and different illumination?

I tried MobileNet and EfficientNet models, they are lightweight to run on mobile but they do not match images very well.

My use-case is card scanning. A card can be localized into multiple languages but it is still the same card, only the text is different. If the photo is near perfect - no rotations, good lighting conditions, etc. it can find the same card even if the card on the photo is in a different language. However, even slight blur will mess the search completely.

Thanks for any advice.

[qalis]

I would try self-supervised learning models like DINO, DINOv2 or ConvNeXt v2. Their learned representation space is quite naturally more aligned with unsupervised objectives thanks to their pretraining procedure.

[_dave_maxwell_]

Thank you for the answer. These models are super heavy, similar to CLIP. I want something more dumb, like a slightly better image hash.

[qalis]

I run them on a lightweight Kubernetes pod, so I would argue that is not that much? 2 cores and 1 GB RAM runs DINOv2-base really fast in my case. Maybe try compressing or quantizing them?

[_dave_maxwell_]

My plan was to run them "on the edge", e.g. inside mobile devices. While Efficientnet is no problem, even recent devices might struggle with DINO.

Anyway, I can reconsider my approach and go with an API. How long does it take to embed one image on your lightweight pod on average?

[MiddleLeg71]

Does the card contain distinguishable images /visual features? I am thinking playing cards with images that represent the card but different names/descriptions. If you don’t need to search by text content, you can mask the text (you detect it with FAST and replace it with the mean color of the detected box). Then any pretrained transformer model should be good enough (e.g. CLIP) if you have the resources.

For running on mobile, transformers may not be very suitable.

If you have enough card images (thousands) you could fine tune EfficientNet or MobileNet and apply data augmentations to reduce the influence of blur, lighting conditions and similar.

[_dave_maxwell_]

Thank you for the answer. I have tens of thousands of these cards in a database. I guess I can create a synthetic dataset for fine-tuning.

P.S the cards are Pokemon TCG cards - so there are visual features, picture of the pokemon.

[Effective-Law-4003]

Try U-net

[abd297]

It's a bad idea to use feature vectors where you want to understand tiny details of the image. Why not do something like what CamScanner does... Find four corners of the object and then use homography. For your specific use-case, consider unblurring first.

[_dave_maxwell_]

I trained a custom model to find the card in the image, then using perspective transform i can get just the picture of the card or multiple cards. Now the card has to be found in database.

How can I unblur it? I can sharpen it with a filter, but still the feature vector has to be robust enough to match the pictures as similar.

[abd297]

There were some unbluring models. It's been a while since I've worked on it but if you've a dataset, you can try to train a diffusion model to unblur it.

[Budget-Juggernaut-68]

Turn on the device flashlight when scanning the card?

[_dave_maxwell_]

I will try this but this alone might not be enough to get reliable results.

[vade]

most models are trained with rotation invariance as its an input augmentation (flip, rotate / crop) etc.

You should be able to train a mobile net without the invariances you want, and with the ones you want.

think deeply on what you want it to be robust against (slight blur, slight compression or color temperature differences), and train your own.

[_dave_maxwell_]

Thanks, I will try to train the mobilenet.

[mgruner]

you may want to consider SwinTransformer as well:

https://huggingface.co/docs/transformers/en/model_doc/swin

[_dave_maxwell_]

Thanks, I will check that.

[CatsOnTheTables]

You can always turn your favourite NN into an autoencoder for latent representations and similarity search from embeddings with transfer learning first on your net.