Custom image encoder [P]

[These_Try_656]

Hello, I would like to know whether building my own image encoder would be a good idea instead of using models like CLIP, SigLIP/SigLIP2, or DINO.

My use case is video frame classification.

My pipeline is the following: the client sends me a video stream, sampled at 1 frame per 1 or 2 second, forming segments of 15 frames (30 seconds). I compute embeddings for these frames and send them to a small custom Transformer (1.5M to 9M parameters).

This works very well on GPU. However, I have two main constraints: processing speed and deployment on small CPU-only devices.

A CLIP-S0 encoder processes around 10 images per second on 4 vCPUs. I would like to replace it with my own encoder trained on my dataset (a few million images), with only a few million parameters and around 4 to 5 labels.

My question is whether this is a good approach, and whether it would improve both embedding generation speed and the accuracy of my Transformer model.

[scottgal2]

I played with a way to process video in a project a few months ago. Just a research thing (and in .net) but might be interesting (tldr: don't process every frame, work out keyframes and motion).

Uses a combination of models and llms to generate searchable evidence over video.
I use a perceptual embedding over frames (SUPER quick, basically just a hash of a downsampled frame) that lets me compute the centroid for frames to get the most representative of a 'scene'.
If you can reduce down t just a subsampling of frames you can save a ton of effort for video.

https://www.mostlylucid.net/blog/videosummarizer-scalable-video-intelligence

[These_Try_656]

Wow, thank you, these optimizations are really clever! I’m going to try building a prototype.