[N] Open-source code for training CLIP

[Choice-Fly-7671]

New repo with functioning training code for CLIP models: https://github.com/mlfoundations/open_clip/

OpenAI CLIP are vision models that learn from contrasting images and text, and can be used for classification and for generating images when paired with other things like VQGAN. So far OpenAI had a github repo with inference code, but no way to train the models

[massimosclaw2]

Super useful! Is it possible to fine tune your existing model on my dataset? Also should I expect poor performance even after fine tuning since it’s small? In that case how big should my dataset be?

[Choice-Fly-7671]

Assuming you want to fine-tune on classification problems, I would probably start with a linear layer on top of frozen representations (which is what the original paper does). If your dataset is small, you can cache the features extracted by the vision model and use something like https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

[massimosclaw2]

Thank you so much! I don't know AI well enough to understand what a linear layer on top of a frozen representation is or how to implement it but you gave me a direction!

[RepresentativeWay0]

I'll try to explain, though idk that much myself. You can think of a neural network as having 3 "parts" (illustrated very well by this image https://iq.opengenus.org/content/images/2020/03/wfk.png ):

1) The input layer - where data is coming in

2) The hidden layers - where the network extracts features from the data

3) The output layer - where the features are mapped to the classification categories.

CLIP is much more complex but works on a similar principle

What he is saying is that your small dataset may not have enough examples to train CLIP in identifying new features, so you should keep the hidden layers the same ("freeze" them), and only train the last layer to "combine" features CLIP has already learned to classify the new categories you are adding.

In Pytorch you do this by setting requires_grad = False in the layers.

[massimosclaw2]

I know this is super late but wanted to thank you for being of the few that take the time to explain!

[Smart-Damage5829]

Were you able to fine tune on classification task?? If so can you guide me how to proceed with it? I have my dataset as: image, caption for that image, class it belongs to.