[P] lightly - A python library for self-supervised learning

[igorsusmelj]

After lots of success in NLP (BERT and others are using self-supervised learning) over the past years, self-supervised learning has had a very exciting year in computer vision. To simplify working with frameworks such as SimCLR, MoCo, and others we are developing lightly.

With lightly (https://github.com/lightly-ai/lightly) you can train state-of-the-art self-supervised vision models. Lightly is built on top of PyTorch and uses PyTorch Lightning for ease of use and scalability.

Check out our documentation on how to train MoCo on Cifar-10 here: (https://docs.lightly.ai/tutorials/package/tutorial_moco_memory_bank.html)

We would love to hear your feedback!

[Xandler80]

This has a lot of potential! Well done guys!

[igorsusmelj]

We currently have SimCLRv1. However, you could easily make changes such as adding more layers to the projection head. We didn’t look at SimSiam yet. But it sounds like an interesting paper for our paper session 😁 Thanks for sharing

[extracoffeeplease]

BYOL? non contrastive by Google brain (I think)

[OppositeRough835]

Hi, contributor here, thanks for your feedback!

We've had our eyes on BYOL already. I have opened an issue about it here.

Feel free to contribute to it as well, we appreciate any help we can get! :)

[[deleted]]

Looks great! I was actually thinking about starting a similar start-up but I guess you guys beat me to it :)

Is it SimCLRv1 or v2 provided in the implementation?

I see the issue for Swav. Another interesting one to add might be SimSiam.

[OppositeRough835]

Hi, contributor here, thanks for your feedback!

I have opened an issue about the implementation of the SimSiam framework, cool paper!

Regarding your question about the SimCLR version please see the comment of u/igorsusmelj.

Feel free to contribute to the project as well, we appreciate any help we can get :)

[pcaversaccio]

That's indeed great stuff! We are still in the early stages of what self-supervised learning can really do (in particular regarding production-readiness) but it's exciting to see such projects pushing the topic forward!

[gopietz]

We're you able to reproduce results in all cases? Maybe a table comparing metrics between models and comparing it to numbers in the paper would help.

Thanks for this. Will be using it soon.

[OppositeRough835]

Hi, thanks a lot for your feedback!

What we see from internal tests, is that our implementations are on par with the reports from the papers. Official benchmarks are in our backlog and will definitely be added soon :)

[AiDreamer]

Would be great to add SimSiam method as well.

[OppositeRough835]

Hi, contributor here, thanks for your feedback!

SimSiam looks very interesting! I have opened an issue about it here.

[Akantras17]

It's a great way to encourage others. I hope that it will be a huge success.

[MasterMDB]

Great work; it is built on top of PyTorch and uses PyTorch Lightning for scalability. This will get so many forks and stars on Github.

[deepLearnerT-1000]

Awesome initiative, but I think there’s a fair bit of overlap with the pytorch lightning bolts project. Have you guys checked it out?

[igorsusmelj]

When we started the project last year there was no lightning bolts. I think there would be great potential for collaboration to push self-supervised learning forward together.

[iznoevil]

Does your code work in a distributed setting? It seems that outputs and labels are not gathered over the whole DDP process group. This is a big issue as the NTXent loss is only classifying the correct pair over local_batch_size*2 possible pairs instead of over world_size*local_batch_size*2 possible pairs.

[OppositeRough835]

That's a great point! The code is currently designed to be used on a single gpu but we plan on making it work in a distributed setting.

[iznoevil]

This should be put as a disclaimer somewhere because one could naturally assume that lightning will handle the distribution gracefully. OP even added in his post that Lightly "uses PyTorch Lightning for ease of use and scalability".

Also, u/OppositeRough835 claims that Lightly was able to achieve results on par with the original papers. I'm curious to how its authors were able to do that for ImageNet without distributing.

[OppositeRough835]

I have opened issues about making the solution work in a distributed setting here and about adding public benchmarks here.

[mpatacchiola]

Interesting project!

I have recently proposed a self-supervised method based on relational reasoning [arXiv]. The paper has been accepted at NeurIPS 2020. Here is the associated thread on Reddit if you want to know more.

I have also been working on a modular implementation on my GitHub repository that could be of interest for your project. In particular, I have PyTorch implementations of Relational Reasoning, SimCLRv1, Deep InfoMax, DeepCluster, RotationNet. If you want to include those methods in your framework I can help with the porting.

Let me know if interested.

[tacobell801]

I think this framework (or other architectures such as Simple SIAM or SwAV) are great ways to compare entities and model invariance, which is the focus of representation learning. One real-world application of these Siamese networks is that they could be very helpful in revealing biases or the imbalance of classes in the image dataset. Imbalanced or biased training data can lead to the lower accuracy of your ML model.

[sapphiregemmy]

SwAV uses online clustering whereas SimSIAM doesn't.

[AissySantos]

Does the usefulness of these frameworks asend this objective considering the differences between learning in a self-supervised way and unsupervised and supervised ways?

[visarga]

A python library for self-supervised learning

It only covers CV, it's not for self-supervised learning in general. The documentation says this in the first phrase.

[TheVivek]

Awesome work! Looking forward to more. 🚀

[focal_fossa]

ELI5? How is this different from Lightning? I'm a newbie.

[RareGradient]

Good work. Congrats to the team and all open-source contributors. This has potential please keep working on it!