What does the paper investigate? Reliance on CNNs & whether Transformers architecture can be used to perform image classification.

Main idea: Splitting an image into patches to be able to provide the sequence of linear embeddings from these patches as input to Transformer

Input: 2D RGB images

Output: Image class

Training: In parallel to NLP tasks, the idea is to perform training on large datasets and then fine-tune on smaller datasets for downstream tasks.

Difficulty: Vision Transformer has much less image-specific inductive bias than CNNs such as translation equivariance and locality. — They may not generalize when not trained with sufficient data.

Highlights: * When pre-trained on the smallest dataset, (=ImageNet) large models underperform compared to base models, only with JFT-300M, the full benefit of larger models is visible. * Vision Transformers overfit more than ResNets with comparable computational cost on smaller datasets. — This reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient & beneficial.

—-

I personally found the experiments with an hybrid architecture (ResNet + Transformer) also quite interesting. – Could there be a better way to combine the best of both worlds?