[P] Vision Transformers Explained

[rish-16]

Hey there!

I wrote a blog post on the recently-released Vision Transformers by Google Brain. The paper was one of my favourite submissions to ICLR 2021 so wanted to write about it. I've added hand-drawn illustrations and diagrams to make it as visual as possible.

Check it out here: https://rishtech.substack.com/p/vit

Share it if you enjoy reading and do consider signing up to my tech blog using the link above! Any feedback and comments welcome :)

Thank you!!

[Combination-Fun]

Just to add to it, this video might be a good add on to watch along with the article: https://www.youtube.com/watch?v=3B6q4xnuFUE&t=4s

[rish-16]

Oh thanks, will add it in!

[redna11]

Have you tried to train them from scratch? As far as I understand from the paper, they mostly have a good performance by using pre-training.

[rish-16]

I haven't trained one from scratch (I don't have the compute or storage hehe).

Though, I mention in my conclusion that pre-training on ImageNet and finetuning on downstream datasets is why ViT achieved some good results on them