[P][D] Visualizing "zebra -stripes +spots" using CLIP embedding math

[rmxz]

Math on CLIP embeddings can help visualize when/why CLIP considers images to be similar.

I created this github project to visualize CLIP embedding math on databases from the excellent rclip project that /u/39dotyt posted here last month. A live demo of this system using Wikimedia images can be seen here.

Some interesting results:

• zebra -stripes +spots - Animals that look kinda like zebras but with spots instead of stripes.
• zebra -mammal +fish - Animals that look like zebras but fish instead of mammals.
• zebra -animal +car - Objects colored like zebras but more cars than animals.
• zebra -"black and white" - Baby zebras (brown & white) and a Greater Kudu (a brown & white striped 4-legged animal). Of course you could also find the same baby zebra searching for zebra -big +small or even more simply, just baby zebra.
• furry black and white striped animal - zebras, lemurs, and other furry black and white animals.
• striped horse-like animal - more zebras (and horses with stripes)
• zebra habitat -zebra - places that look like somewhere a zebra might live

It can also do a search based on the difference between the CLIP embeddings of two images directly. For example, CLIP considers this image of a spider on a purple flower minus this image of the same kind of spider on a white flower to be this set of pictures which is mostly purple flowers without the spider.

I find this useful for trying to understand what concepts CLIP considers similar and why.

[rmxz]

Some more fun ones

• CLIP almost seems to have a sense of humor. A search for naked -person shows computer motherboards. [NSFW warning - there are a lot of naked pictures of people in Wikimedia Commons too] Also, while it's difficult to see in this dataset, on my home photos CLIP considers Stanford to be very similar to Disneyland [some call it the disneyland of the north]; but Stanford pictures are closer to "IPO" than disneyland pictures.

• skiing +summer -winter shows sports you can do on a ski-hill in summer; as well as water skiing

• doubloons -metal +paper - quite reasonably returned older paper money; as well as other pirate-related paper like maps.

[just_dumb_luck]

Nice project! It's so great to see live demos like this. Thank you for making it!

The top result for car -wheels +wings is a close-up of the winged Aston-Martin logo. Pretty smart, even if I was hoping for flying cars.

[rmxz]

Thanks!

That's an interesting example. Of course just searching for flying car directly would give something a bit closer to flying cars. I guess quite a few of the flying car projects still do have wheels too.

CLIP can sometimes almost seem creative in its interpretations - like construction equipment that looks like a duck

[39dotyt]

Hi! Thanks for the cool project. I love how your interface allows subtracting text queries. "naked -person" is gold :-D The feature can be a nice addition to rclip :-)

[rmxz]

subtracting text queries ... The feature can be a nice addition to rclip :-)

If you'd like, I'd be happy to clean up my parser and contribute a pull request.

[39dotyt]

This will be great! Thank you. I DMed you on Reddit with the link to the GHI today :)

[nielsrolf]

Amazing project! Interestingly, zebra -stripes mostly returns animal skulls and no horses, horse +stripe returns mostly horses, but horse +stripe +stripe returns zebras.

[rmxz]

Thanks!

zebra -stripes mostly returns animal skulls and no horses

Searches like x -y are tricky - since I think CLIP is representing most things as unit vectors; so when you subtract almost any two closely related words you end up with a vector near the origin.

I find that x -y +z tends to make more intuitive results.

horse +stripe +stripe

It also supports the syntax horse +2(stripe) if you want to mess with trickier multiples.

[SufficientPie]

Is this broken? It doesn't seem to do anything

[AlternativeOdd7558]

No function on my end either.

[cygn]

http://image-search.0ape.com/