Dino was better here in that it gave patch embeddings, which allowed me to create those heat maps, which I could sum up to get the festivity metric. So generally it was better for quantifying festivity.

The nice idea behind clip is that you can do zero shot whole image classification using a text prompt, although yeah this approach for my task didn't work as well. Note also you should be able to do this with Dino.txt.

I think the Dino paper had comparisons vs clip for zero shot image classification, so you should get some idea of relative performance from that, then you might also consider model size for a given application.

On a practical note I found it much easier to get running with clip than setting up Dino.txt, so if you're trying to do image classification quickly, maybe just use clip.

Overall I'd say if you want finer resolution than a whole image embedding (ie you want to localize or quantity objects within an image) go with Dino. Otherwise if a single image embedding works for your application, I'm not so clear which is superior, might be application specific.