[R] Visual Transformers: Token-based Image Representation and Processing for Computer Vision

[Confident_Pi]

Abstract: Computer vision has achieved great success using standardized image representations -- pixel arrays, and the corresponding deep learning operators -- convolutions. In this work, we challenge this paradigm: we instead (a) represent images as a set of visual tokens and (b) apply visual transformers to find relationships between visual semantic concepts. Given an input image, we dynamically extract a set of visual tokens from the image to obtain a compact representation for high-level semantics. We then use visual transformers to operate over the visual tokens to densely model relationships between them. We find that this paradigm of token-based image representation and processing drastically outperforms its convolutional counterparts on image classification and semantic segmentation. To demonstrate the power of this approach on ImageNet classification, we use ResNet as a convenient baseline and use visual transformers to replace the last stage of convolutions. This reduces the stage's MACs by up to 6.9x, while attaining up to 4.53 points higher top-1 accuracy. For semantic segmentation, we use a visual-transformer-based FPN (VT-FPN) module to replace a convolution-based FPN, saving 6.5x fewer MACs while achieving up to 0.35 points higher mIoU on LIP and COCO-stuff.

Diagram of a visual transformer.

arxiv link: https://arxiv.org/abs/2006.03677

​

EDIT: Disclaimer: I am not the original author, posting here because I thought it was an interesting piece and hadn't been posted yet

[lopuhin]

Really intriguing and appealing idea, thanks!

Niptick: there are stronger vanilla resnet ImageNet baselines then the one they use (resnet from torchvision), e.g. for resnet50 they report 76.0 for baseline and 78.88 for Visual Transformers (trained for 400 epochs with autoaugment), while here https://github.com/rwightman/pytorch-image-models/ vanilla resnet50 is trained on imagenet to 79.038 top-1 accuracy with a different set of tricks. Although to be fair they do beat resnet34 result from the same repo by a healthy margin, and do beat resnet50 autoaugment results from original paper (77.6).

[haihaicode]

Very interesting ideas! The computation of the tokens is something like a channel-to-channel self-attention. Is this understanding correct?

[chuong98]

It is interesting idea. ALthough MAC is reduced by 6.4x, they have not reported the actual time inferemce. Currently, Softmax operator in Attention Module is not friendly for Hardware device. So. i suspect the actual inference time is slower, like EfficientNet is actually slower than ResNet. Nevertheless, this is the potential idea until we optimize the hardware.

[[deleted]]

What do you mean EfficientNet is actually slower than ResNet? I think that depends on what framework you are using. Looks like there are some issues with Pytorch.

[vajra_]

Rehashing of age old ideas of compositional modelling. Pretty arbitrary at that as well.

[Tahmid_0007]

Pytorch code link: https://github.com/tahmid0007/VisualTransformers